Jul 17 00:16:45.304265 ip-10-0-139-41 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 00:16:45.697595 ip-10-0-139-41 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 00:16:45.697595 ip-10-0-139-41 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 00:16:45.697595 ip-10-0-139-41 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 00:16:45.697595 ip-10-0-139-41 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 00:16:45.697595 ip-10-0-139-41 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 00:16:45.697595 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.691135    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698005    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698024    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698029    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698034    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698039    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698044    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698048    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698051    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698055    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698059    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698064    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698069    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698073    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698076    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698080    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698084    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698088    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698091    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698095    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 00:16:45.699466 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698109    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698114    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698118    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698122    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698126    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698130    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698134    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698138    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698142    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698148    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698153    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698157    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698161    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698165    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698170    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698174    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698179    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698184    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698188    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698193    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 00:16:45.712456 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698197    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698204    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698212    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698216    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698221    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698226    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698230    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698236    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698243    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698249    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698254    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698260    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698265    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698270    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698275    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698279    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698284    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698288    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698294    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698299    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 00:16:45.714241 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698303    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698307    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698314    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698320    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698324    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698331    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698336    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698361    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698365    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698369    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698374    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698378    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698384    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698388    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698392    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698396    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698400    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698403    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698407    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 00:16:45.716453 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698411    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698415    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698419    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698423    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698428    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698432    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698436    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698440    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698444    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698448    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698451    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698455    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698459    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698464    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698468    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698472    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698477    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698481    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698486    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 00:16:45.732041 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698490    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.698494    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699393    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699401    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699405    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699408    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699411    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699416    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699419    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699422    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699425    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699428    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699431    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699434    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699437    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699440    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699442    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699446    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699450    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699453    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 00:16:45.734449 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699456    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699461    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699465    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699468    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699471    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699474    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699479    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699482    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699485    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699488    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699491    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699494    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699496    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699499    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699502    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699505    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699508    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699511    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699514    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699517    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 00:16:45.736334 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699520    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699523    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699525    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699532    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699535    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699538    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699541    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699543    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699546    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699548    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699551    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699554    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699557    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699560    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699563    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699566    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699568    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699571    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699574    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699576    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 00:16:45.738660 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699579    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699581    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699584    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699586    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699589    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699591    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699594    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699596    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699598    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699601    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699604    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699606    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699609    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699616    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699619    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699624    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699626    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699629    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699631    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699634    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 00:16:45.751938 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699636    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699639    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699642    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699645    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699648    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699651    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699653    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699658    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699660    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699663    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699666    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699668    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699671    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699676    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699679    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699683    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699685    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699689    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699691    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 00:16:45.754300 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699694    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.699696    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700168    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700176    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700179    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700183    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700188    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700192    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700197    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700200    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700203    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700206    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700208    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700211    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700214    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700217    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700220    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700223    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700225    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700228    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 00:16:45.973940 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700231    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 00:16:45.767930 ip-10-0-139-41 systemd[1]: Started Kubernetes Kubelet.
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700233    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700236    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700239    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700242    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700244    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700248    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700252    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700254    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700257    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700260    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700262    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700265    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700267    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700270    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700272    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700275    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700277    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700284    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 00:16:45.985843 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700287    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700292    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700294    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700297    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700300    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700302    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700305    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700307    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700310    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700312    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700315    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700318    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700321    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700323    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700326    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700329    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700332    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700334    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700337    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700339    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 00:16:45.990039 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700342    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700344    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700347    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700349    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700352    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700354    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700357    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700359    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700362    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700364    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700367    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700371    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700373    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700378    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700380    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700383    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700386    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700389    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700392    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700394    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 00:16:45.990734 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700399    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700402    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700405    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700408    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700411    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700413    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700416    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700419    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700421    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700424    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700426    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700429    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700432    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700434    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700437    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700440    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700443    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700446    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700448    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700450    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 00:16:45.991415 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700453    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.700455    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700531    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700545    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700553    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700558    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700563    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700567    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700571    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700576    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700579    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700582    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700586    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700589    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700592    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700595    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700598    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700602    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700605    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700608    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700612    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700615    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700619    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700622    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 00:16:45.992060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700627    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700630    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700633    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700637    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700640    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700644    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700647    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700650    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700653    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700656    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700660    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700665    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700668    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700672    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700675    2573 flags.go:64] FLAG: --enable-server="true"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700679    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700684    2573 flags.go:64] FLAG: --event-burst="100"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700687    2573 flags.go:64] FLAG: --event-qps="50"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700690    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700694    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700697    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700701    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700704    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700707    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700710    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 17 00:16:45.992774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700713    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700716    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700719    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700722    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700725    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700728    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700731    2573 flags.go:64] FLAG: --feature-gates=""
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700735    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700738    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700741    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700744    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700747    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700750    2573 flags.go:64] FLAG: --help="false"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700753    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-139-41.ec2.internal"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700756    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700759    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700762    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700766    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700771    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700774    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700778    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700781    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700784    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 00:16:45.993546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700804    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700808    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700811    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700815    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700818    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700821    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700824    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700827    2573 flags.go:64] FLAG: --lock-file=""
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700830    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700833    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700836    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700842    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700845    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700848    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700850    2573 flags.go:64] FLAG: --logging-format="text"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700853    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700857    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700860    2573 flags.go:64] FLAG: --manifest-url=""
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700862    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700867    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700870    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700874    2573 flags.go:64] FLAG: --max-pods="110"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700877    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700880    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700883    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 00:16:45.994259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700891    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700895    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700900    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700903    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700912    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700916    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700919    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700922    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700926    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700933    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700936    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700939    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700942    2573 flags.go:64] FLAG: --port="10250"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700945    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700948    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0532333e9f5fbf434"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700951    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700954    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700957    2573 flags.go:64] FLAG: --register-node="true"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700960    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700964    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700967    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700969    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700972    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700976    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700979    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 00:16:45.995053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700982    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700985    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700988    2573 flags.go:64] FLAG: --runonce="false"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700991    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700994    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700996    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.700999    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701002    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701005    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701010    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701013    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701016    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701021    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701024    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701027    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701030    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701034    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701036    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701042    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701045    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701048    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701052    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701055    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701058    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701061    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 00:16:45.995761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701064    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701067    2573 flags.go:64] FLAG: --v="2"
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701072    2573 flags.go:64] FLAG: --version="false"
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701079    2573 flags.go:64] FLAG: --vmodule=""
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701083    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701086    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701214    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701218    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701221    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701225    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701229    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701232    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701235    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701237    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701241    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701245    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701250    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701254    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701257    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701262    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 00:16:45.996534 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701265    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701268    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701271    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701274    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701277    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701279    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701282    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701285    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701288    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701290    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701293    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701296    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701299    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701302    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701304    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701307    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701309    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701312    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701315    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701317    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 00:16:45.997276 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701320    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701322    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701325    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701328    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701330    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701333    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701336    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701338    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701342    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701345    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701348    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701352    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701355    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701358    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701361    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701363    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701366    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701370    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701373    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 00:16:45.997970 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701375    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701378    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701381    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701383    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701387    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701389    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701392    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701394    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701397    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701400    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701402    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701405    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701407    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701410    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701412    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701415    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701418    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701420    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701423    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701425    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 00:16:45.998491 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701428    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701432    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701435    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701437    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701442    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701445    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701447    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701450    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701453    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701455    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701458    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701462    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701465    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701467    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701470    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701473    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701475    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701478    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701481    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701483    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 00:16:45.999085 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701485    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701488    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701491    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701493    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701496    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.701499    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.701974    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.709326    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.709345    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.709361    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.709367    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709426    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709431    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709437    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709441    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 00:16:46.165275 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709444    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709448    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709453    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709456    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709459    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709462    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709464    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709467    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709471    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709474    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709476    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709479    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709482    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709485    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709487    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709491    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709494    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709496    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709501    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 00:16:46.166299 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709506    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709510    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709513    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709515    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709518    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709521    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709523    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709526    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709528    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709531    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709534    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709539    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709542    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709545    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709547    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709551    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709554    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709556    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709559    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709562    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 00:16:46.166953 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709564    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709567    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709570    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709573    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709576    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709579    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709582    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709584    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709587    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709589    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709592    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709594    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709597    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709600    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709602    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709605    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709608    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709610    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709612    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709615    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 00:16:46.167526 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709617    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709620    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709623    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709626    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709629    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709632    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709634    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709639    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709642    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709645    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709647    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709650    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709652    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709655    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709658    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709660    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709663    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709666    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709668    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709671    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 00:16:46.168100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709673    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709676    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709679    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709681    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709684    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709687    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709689    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709693    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709696    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709700    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709702    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709705    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709708    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709711    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709714    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 00:16:46.168668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709718    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.709723    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709864    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709869    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709872    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709877    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709879    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709882    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709885    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709887    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709890    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709893    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709895    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709903    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709906    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 00:16:46.169433 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709909    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709912    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709914    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709917    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709920    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709922    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709925    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709927    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709930    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709932    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709935    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709938    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709940    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709942    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709945    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709948    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709951    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709955    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709958    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 00:16:46.169994 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709960    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709963    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709966    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709970    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709975    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709977    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709980    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709983    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709985    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709988    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709991    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709994    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.709997    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710000    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710003    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710006    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710008    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710011    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710013    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 00:16:46.308412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710016    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710019    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710021    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710024    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710026    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710029    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710031    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710034    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710036    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710039    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710042    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710046    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710050    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710053    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710056    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710059    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710063    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710066    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710069    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 00:16:46.309217 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710072    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710075    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710078    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710081    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710083    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710087    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710089    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710092    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710095    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710097    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710100    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710102    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710105    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710108    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710110    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710113    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710115    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710118    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710120    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710122    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 00:16:46.310038 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710126    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710130    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710132    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710135    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710139    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710142    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710145    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710147    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:45.710150    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.710157    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.710296    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.712298    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.713482    2573 server.go:1019] "Starting client certificate rotation"
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.713601    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 00:16:46.310647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.713662    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.717188    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.719005    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.732399    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.736916    2573 log.go:25] "Validated CRI v1 image API"
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.737857    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.741853    2573 fs.go:135] Filesystem UUIDs: map[1368ad9f-64f7-4c1d-b1ee-23545da71dd1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7c443732-88ed-4818-aef9-942210c3cbbd:/dev/nvme0n1p3]
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.741871    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 00:16:46.311266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.742453    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.747059    2573 manager.go:216] Machine: {Timestamp:2026-07-17 00:16:45.745672773 +0000 UTC m=+0.321825338 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118168 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d9f0c8942d192f294c258778ffb0 SystemUUID:ec23d9f0-c894-2d19-2f29-4c258778ffb0 BootID:fb4243d3-8dfc-42b4-ab92-8b7719192e42 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:83:22:47:a6:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:83:22:47:a6:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:65:c7:ad:19:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.747172    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.747265    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.748827    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.748859    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-41.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.750142    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.750152    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.750165    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.750805    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.751995    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.754832    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.754993    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.755043    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.755062    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.755090    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.755105    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.755123    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 00:16:46.312075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.755692    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.755721    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.759080    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.761549    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.761577    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762630    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762644    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762650    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762656    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762664    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762670    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762679    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762685    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762692    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762697    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762706    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.762711    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.763356    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.763364    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.763814    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w2j5m"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.765171    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-41.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.765714    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.765719    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.766863    2573 server.go:1268] "Started kubelet"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.766943    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.766959    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.767036    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.768589    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.769454    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.770938    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w2j5m"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.775415    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.775741    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.775750    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.775865    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776391    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776430    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776551    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776560    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776693    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776708    2573 factory.go:55] Registering systemd factory
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776718    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776777    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776962    2573 factory.go:153] Registering CRI-O factory
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.776978    2573 factory.go:223] Registration of the crio container factory successfully
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.777007    2573 factory.go:103] Registering Raw factory
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.777020    2573 manager.go:1196] Started watching for new ooms in manager
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.777446    2573 manager.go:349] Starting recovery of all containers
Jul 17 00:16:46.312648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.782189    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.786313    2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-41.ec2.internal\" not found" node="ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.792826    2573 manager.go:354] Recovery completed
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.797227    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.799225    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.799255    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.799269    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.799751    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.799760    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.799778    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.799818    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.800493    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.800520    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.800530    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.800921    2573 policy_none.go:49] "None policy: Start"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.800934    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.800944    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.801990    2573 policy_none.go:47] "Start"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.845428    2573 manager.go:337] "Starting Device Plugin manager"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.845472    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.845482    2573 server.go:87] "Starting device plugin registration server"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.845811    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.845827    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.846040    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.846126    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.846138    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.846578    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.846629    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.946716    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.947909    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.947946    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.947961    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.948005    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.957849    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.957874    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-41.ec2.internal\": node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.960120    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.961585    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.961622    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.961643    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.961675    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:45.963997    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:45.972922    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.062268    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"]
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.062551    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.064286    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.064316    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.064327    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Jul 17 00:16:46.313873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.065812    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.065911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.065940    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.066627    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.066651    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.066661    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.067474    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.067504    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.067518    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.068603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.068629    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.071530    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.071562    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.071577    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.073194    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.078841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/002d47e5af4c93f026b6cc20e8b1a908-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"002d47e5af4c93f026b6cc20e8b1a908\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.078865    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ea6cd52e772b8df4e90080acb99b082-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"0ea6cd52e772b8df4e90080acb99b082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.078889    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ea6cd52e772b8df4e90080acb99b082-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"0ea6cd52e772b8df4e90080acb99b082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.098771    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-41.ec2.internal\" not found" node="ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.103979    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-41.ec2.internal\" not found" node="ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.173647    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.179194    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ea6cd52e772b8df4e90080acb99b082-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"0ea6cd52e772b8df4e90080acb99b082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.179229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ea6cd52e772b8df4e90080acb99b082-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"0ea6cd52e772b8df4e90080acb99b082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.179247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/002d47e5af4c93f026b6cc20e8b1a908-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"002d47e5af4c93f026b6cc20e8b1a908\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.179283    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ea6cd52e772b8df4e90080acb99b082-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"0ea6cd52e772b8df4e90080acb99b082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.179281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ea6cd52e772b8df4e90080acb99b082-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"0ea6cd52e772b8df4e90080acb99b082\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315888 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.179371    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/002d47e5af4c93f026b6cc20e8b1a908-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"002d47e5af4c93f026b6cc20e8b1a908\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.315888 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.274729    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.447155 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.375308    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.447155 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.401771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.447155 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.407875    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:46.592242 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.475583    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.592242 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.576216    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.728620 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.676615    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.728620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.713209    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 00:16:46.728620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.713388    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 00:16:46.728620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.713433    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 00:16:46.863078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.775591    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 00:11:45 +0000 UTC" deadline="2027-12-23 09:48:41.410594622 +0000 UTC"
Jul 17 00:16:46.863078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.775615    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12585h31m54.634981588s"
Jul 17 00:16:46.863078 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.777118    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:46.863078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:46.806536    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 00:16:47.006247 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.877161    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:47.006247 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:46.977686    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:47.143641 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:47.078082    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:47.284742 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:47.178480    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Jul 17 00:16:47.284742 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.244995    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 00:16:47.284742 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.276384    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:47.286121 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.285163    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 00:16:47.288045 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.286583    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Jul 17 00:16:47.433310 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.291268    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.757825    2573 apiserver.go:52] "Watching apiserver"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.762943    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.764261    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pfsxc","kube-system/konnectivity-agent-zxwfp","kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx","openshift-cluster-node-tuning-operator/tuned-7r58z","openshift-dns/node-resolver-6x662","openshift-image-registry/node-ca-rxcmn","openshift-multus/multus-additional-cni-plugins-k8pdf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal","openshift-multus/multus-cdpqd","openshift-multus/network-metrics-daemon-vjwrp","openshift-network-diagnostics/network-check-target-7nrcr","openshift-network-operator/iptables-alerter-6w8v5","openshift-ovn-kubernetes/ovnkube-node-645g2"]
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.765901    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.767660    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sqd24\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.767878    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.768048    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.768200    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.770178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.770699    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.771508    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.771729    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zx5c2\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.771908    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.772104    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.772240    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.772934    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.773131    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.773256    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.773413    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5td26\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.773571    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.773733    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.774151    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.774365    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.774501    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.774671    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4cmnz\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.774863    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.775122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.775553    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.776083    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.776269    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:47.776334    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.777221    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 00:16:47.799833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.777433    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.777649    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dj9sv\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.777801    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.777884    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778032    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbctf\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778141    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v6vs\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778152    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778275    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778389    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778485    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778536    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778606    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.778776    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.786690    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.787226    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.787353    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g8c9m\""
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.787499    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.787837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovn-node-metrics-cert\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.787888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-etc-selinux\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.787962    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gxq\" (UniqueName: \"kubernetes.io/projected/88a18b2f-8553-417d-893b-64d5fc11fc66-kube-api-access-s7gxq\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788015    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-lib-modules\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8df807-8d49-4d3b-93de-027f715654d0-tmp\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-etc-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788167    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovnkube-config\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a2ee30-6f7d-438c-bac7-52add958ae4c-tmp-dir\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.803035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fb7\" (UniqueName: \"kubernetes.io/projected/03a2ee30-6f7d-438c-bac7-52add958ae4c-kube-api-access-h5fb7\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788284    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-socket-dir-parent\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788320    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f892710-cf20-4bdb-aa0b-0c42e8f9436a-konnectivity-ca\") pod \"konnectivity-agent-zxwfp\" (UID: \"7f892710-cf20-4bdb-aa0b-0c42e8f9436a\") " pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788338    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-cnibin\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-cni-binary-copy\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788407    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-cni-bin\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-conf-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckv7p\" (UniqueName: \"kubernetes.io/projected/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-kube-api-access-ckv7p\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-kubernetes\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-var-lib-kubelet\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovnkube-script-lib\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-netns\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tgn\" (UniqueName: \"kubernetes.io/projected/bb8df807-8d49-4d3b-93de-027f715654d0-kube-api-access-h7tgn\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-sys-fs\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-os-release\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788863    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zgtg\" (UniqueName: \"kubernetes.io/projected/00c347f5-1f2b-402c-8edf-cc04d4832887-kube-api-access-2zgtg\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9896195f-f58d-4c3f-8a1c-c486d1293cc4-iptables-alerter-script\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.803829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.788943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-node-log\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6kz\" (UniqueName: \"kubernetes.io/projected/983c692a-3a52-4a5d-b36f-4ded907ffd0d-kube-api-access-dd6kz\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-ovn\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789166    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-k8s-cni-cncf-io\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28d955d-d569-4155-8732-247fd315be8f-host\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789220    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f892710-cf20-4bdb-aa0b-0c42e8f9436a-agent-certs\") pod \"konnectivity-agent-zxwfp\" (UID: \"7f892710-cf20-4bdb-aa0b-0c42e8f9436a\") " pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-cni-multus\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-kubelet\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d28d955d-d569-4155-8732-247fd315be8f-serviceca\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789350    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-device-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-run-netns\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-daemon-config\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.868928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-multus-certs\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-etc-kubernetes\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789533    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789645    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jfn9h\""
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:47.789628    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysctl-d\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysctl-conf\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789904    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-os-release\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-kubelet\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.789997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-systemd\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-log-socket\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-registration-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-systemd-units\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790379    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-var-lib-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-cni-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-hostroot\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.869970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790421    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysconfig\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9896195f-f58d-4c3f-8a1c-c486d1293cc4-host-slash\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-cni-bin\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-socket-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790554    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03a2ee30-6f7d-438c-bac7-52add958ae4c-hosts-file\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-env-overrides\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlt7m\" (UniqueName: \"kubernetes.io/projected/d28d955d-d569-4155-8732-247fd315be8f-kube-api-access-vlt7m\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-modprobe-d\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790646    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-systemd\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-slash\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-system-cni-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-run\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-system-cni-dir\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-cnibin\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790820    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.870486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb595638-7449-478a-8505-2e0e531829a1-kubelet-config\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb595638-7449-478a-8505-2e0e531829a1-dbus\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-sys\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-host\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb8df807-8d49-4d3b-93de-027f715654d0-etc-tuned\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.790998    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56kl\" (UniqueName: \"kubernetes.io/projected/9896195f-f58d-4c3f-8a1c-c486d1293cc4-kube-api-access-n56kl\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.791022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-cni-netd\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.791118    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:47.791166    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.794506    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.811713    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bbfbw"
Jul 17 00:16:47.871104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.818257    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bbfbw"
Jul 17 00:16:47.875963 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:47.875922    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea6cd52e772b8df4e90080acb99b082.slice/crio-858bfafa5f0a29c36fecc0c26abe8399e36d2dd05b3bb460a90ce2da503aa5bb WatchSource:0}: Error finding container 858bfafa5f0a29c36fecc0c26abe8399e36d2dd05b3bb460a90ce2da503aa5bb: Status 404 returned error can't find the container with id 858bfafa5f0a29c36fecc0c26abe8399e36d2dd05b3bb460a90ce2da503aa5bb
Jul 17 00:16:47.876100 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:47.876084    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002d47e5af4c93f026b6cc20e8b1a908.slice/crio-250c61b58f59ce2e8d72240f37dfd3349599e517ebeb498f312201a98dcafea3 WatchSource:0}: Error finding container 250c61b58f59ce2e8d72240f37dfd3349599e517ebeb498f312201a98dcafea3: Status 404 returned error can't find the container with id 250c61b58f59ce2e8d72240f37dfd3349599e517ebeb498f312201a98dcafea3
Jul 17 00:16:47.877387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.877368    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 00:16:47.886661 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.886635    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 00:16:47.891271 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.891361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-multus-certs\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.891361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-etc-kubernetes\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.891361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:47.891361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysctl-d\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysctl-conf\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891369    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891376    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-multus-certs\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-os-release\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-etc-kubernetes\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-kubelet\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-systemd\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-log-socket\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-registration-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysctl-d\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891554    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysctl-conf\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891571    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-systemd\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-systemd-units\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891597    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-log-socket\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891602    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-kubelet\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891518    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-os-release\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891611    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-var-lib-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-registration-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-cni-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-hostroot\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysconfig\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9896195f-f58d-4c3f-8a1c-c486d1293cc4-host-slash\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891646    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-systemd-units\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-cni-bin\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.891955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891759    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891770    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9896195f-f58d-4c3f-8a1c-c486d1293cc4-host-slash\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-cni-bin\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-socket-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03a2ee30-6f7d-438c-bac7-52add958ae4c-hosts-file\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-hostroot\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-env-overrides\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891870    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-cni-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlt7m\" (UniqueName: \"kubernetes.io/projected/d28d955d-d569-4155-8732-247fd315be8f-kube-api-access-vlt7m\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891910    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-sysconfig\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-modprobe-d\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891933    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03a2ee30-6f7d-438c-bac7-52add958ae4c-hosts-file\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891898    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-socket-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.891954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-systemd\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-slash\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-system-cni-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-modprobe-d\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-var-lib-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.892717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-run\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-system-cni-dir\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-cnibin\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892141    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892172    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-systemd\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892183    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-system-cni-dir\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892194    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-system-cni-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892219    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-slash\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892256    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb595638-7449-478a-8505-2e0e531829a1-kubelet-config\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb595638-7449-478a-8505-2e0e531829a1-dbus\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892310    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-sys\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-env-overrides\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-host\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.893509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892375    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb595638-7449-478a-8505-2e0e531829a1-kubelet-config\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb8df807-8d49-4d3b-93de-027f715654d0-etc-tuned\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892399    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb595638-7449-478a-8505-2e0e531829a1-dbus\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892418    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-sys\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892429    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-host\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892447    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56kl\" (UniqueName: \"kubernetes.io/projected/9896195f-f58d-4c3f-8a1c-c486d1293cc4-kube-api-access-n56kl\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892471    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00c347f5-1f2b-402c-8edf-cc04d4832887-cnibin\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-cni-netd\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892540    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-cni-netd\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892341    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-run\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovn-node-metrics-cert\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-etc-selinux\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gxq\" (UniqueName: \"kubernetes.io/projected/88a18b2f-8553-417d-893b-64d5fc11fc66-kube-api-access-s7gxq\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-lib-modules\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892699    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8df807-8d49-4d3b-93de-027f715654d0-tmp\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-etc-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.894174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovnkube-config\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a2ee30-6f7d-438c-bac7-52add958ae4c-tmp-dir\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fb7\" (UniqueName: \"kubernetes.io/projected/03a2ee30-6f7d-438c-bac7-52add958ae4c-kube-api-access-h5fb7\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-socket-dir-parent\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f892710-cf20-4bdb-aa0b-0c42e8f9436a-konnectivity-ca\") pod \"konnectivity-agent-zxwfp\" (UID: \"7f892710-cf20-4bdb-aa0b-0c42e8f9436a\") " pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-cnibin\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.892966    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-cni-binary-copy\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.893311    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a2ee30-6f7d-438c-bac7-52add958ae4c-tmp-dir\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.894896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.893809    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-etc-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.898247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.893047    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-etc-selinux\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.898944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-cni-bin\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.898944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898276    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.898944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898366    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-cnibin\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.898944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898609    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-lib-modules\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.898944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898660    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-cni-bin\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.898944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-socket-dir-parent\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.898944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898927    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-cni-binary-copy\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.898978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-conf-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckv7p\" (UniqueName: \"kubernetes.io/projected/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-kube-api-access-ckv7p\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-kubernetes\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-var-lib-kubelet\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-conf-dir\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovnkube-script-lib\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899123    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-netns\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjnd\" (UniqueName: \"kubernetes.io/projected/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-kube-api-access-nbjnd\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899172    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tgn\" (UniqueName: \"kubernetes.io/projected/bb8df807-8d49-4d3b-93de-027f715654d0-kube-api-access-h7tgn\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-sys-fs\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899214    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-os-release\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.899555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zgtg\" (UniqueName: \"kubernetes.io/projected/00c347f5-1f2b-402c-8edf-cc04d4832887-kube-api-access-2zgtg\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.900557 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.900315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9896195f-f58d-4c3f-8a1c-c486d1293cc4-iptables-alerter-script\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.900557 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.900399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-node-log\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.901221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:47.901299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901239    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8df807-8d49-4d3b-93de-027f715654d0-tmp\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.901299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6kz\" (UniqueName: \"kubernetes.io/projected/983c692a-3a52-4a5d-b36f-4ded907ffd0d-kube-api-access-dd6kz\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.901299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901243    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-netns\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-ovn\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901390    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-k8s-cni-cncf-io\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28d955d-d569-4155-8732-247fd315be8f-host\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f892710-cf20-4bdb-aa0b-0c42e8f9436a-agent-certs\") pod \"konnectivity-agent-zxwfp\" (UID: \"7f892710-cf20-4bdb-aa0b-0c42e8f9436a\") " pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-cni-multus\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-kubelet\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901561    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d28d955d-d569-4155-8732-247fd315be8f-serviceca\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901659    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovn-node-metrics-cert\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-device-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovnkube-script-lib\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901802    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-device-dir\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-run-netns\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901847    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-daemon-config\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901852    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-openvswitch\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901860    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88a18b2f-8553-417d-893b-64d5fc11fc66-sys-fs\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.899244    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-etc-kubernetes\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901981    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-host-run-netns\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.902076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb8df807-8d49-4d3b-93de-027f715654d0-etc-tuned\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.902117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/983c692a-3a52-4a5d-b36f-4ded907ffd0d-ovnkube-config\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.902501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-multus-daemon-config\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.901192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9896195f-f58d-4c3f-8a1c-c486d1293cc4-iptables-alerter-script\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.902642    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00c347f5-1f2b-402c-8edf-cc04d4832887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:47.902766    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:47.902905    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret podName:fb595638-7449-478a-8505-2e0e531829a1 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:48.402845963 +0000 UTC m=+2.978998525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret") pod "global-pull-secret-syncer-pfsxc" (UID: "fb595638-7449-478a-8505-2e0e531829a1") : object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.902978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8df807-8d49-4d3b-93de-027f715654d0-var-lib-kubelet\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.902980    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28d955d-d569-4155-8732-247fd315be8f-host\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-kubelet\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903047    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-run-k8s-cni-cncf-io\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903084    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-run-ovn\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.905928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d28d955d-d569-4155-8732-247fd315be8f-serviceca\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.906635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903176    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-host-var-lib-cni-multus\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.906635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903347    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-os-release\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.906635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903394    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/983c692a-3a52-4a5d-b36f-4ded907ffd0d-node-log\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.906635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.903424    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f892710-cf20-4bdb-aa0b-0c42e8f9436a-konnectivity-ca\") pod \"konnectivity-agent-zxwfp\" (UID: \"7f892710-cf20-4bdb-aa0b-0c42e8f9436a\") " pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.906635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.906215    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlt7m\" (UniqueName: \"kubernetes.io/projected/d28d955d-d569-4155-8732-247fd315be8f-kube-api-access-vlt7m\") pod \"node-ca-rxcmn\" (UID: \"d28d955d-d569-4155-8732-247fd315be8f\") " pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:47.907040 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.907007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f892710-cf20-4bdb-aa0b-0c42e8f9436a-agent-certs\") pod \"konnectivity-agent-zxwfp\" (UID: \"7f892710-cf20-4bdb-aa0b-0c42e8f9436a\") " pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.913324 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.913266    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56kl\" (UniqueName: \"kubernetes.io/projected/9896195f-f58d-4c3f-8a1c-c486d1293cc4-kube-api-access-n56kl\") pod \"iptables-alerter-6w8v5\" (UID: \"9896195f-f58d-4c3f-8a1c-c486d1293cc4\") " pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:47.916036 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.915980    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fb7\" (UniqueName: \"kubernetes.io/projected/03a2ee30-6f7d-438c-bac7-52add958ae4c-kube-api-access-h5fb7\") pod \"node-resolver-6x662\" (UID: \"03a2ee30-6f7d-438c-bac7-52add958ae4c\") " pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.918103 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.917623    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tgn\" (UniqueName: \"kubernetes.io/projected/bb8df807-8d49-4d3b-93de-027f715654d0-kube-api-access-h7tgn\") pod \"tuned-7r58z\" (UID: \"bb8df807-8d49-4d3b-93de-027f715654d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.918563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.918526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zgtg\" (UniqueName: \"kubernetes.io/projected/00c347f5-1f2b-402c-8edf-cc04d4832887-kube-api-access-2zgtg\") pod \"multus-additional-cni-plugins-k8pdf\" (UID: \"00c347f5-1f2b-402c-8edf-cc04d4832887\") " pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.919164 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.919134    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6kz\" (UniqueName: \"kubernetes.io/projected/983c692a-3a52-4a5d-b36f-4ded907ffd0d-kube-api-access-dd6kz\") pod \"ovnkube-node-645g2\" (UID: \"983c692a-3a52-4a5d-b36f-4ded907ffd0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:47.919266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.919134    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckv7p\" (UniqueName: \"kubernetes.io/projected/9662c6e8-79c3-4f03-b1bf-e9f85cac2207-kube-api-access-ckv7p\") pod \"multus-cdpqd\" (UID: \"9662c6e8-79c3-4f03-b1bf-e9f85cac2207\") " pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.919510 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.919492    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gxq\" (UniqueName: \"kubernetes.io/projected/88a18b2f-8553-417d-893b-64d5fc11fc66-kube-api-access-s7gxq\") pod \"aws-ebs-csi-driver-node-zd5vx\" (UID: \"88a18b2f-8553-417d-893b-64d5fc11fc66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:47.919867 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.919851    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7r58z"
Jul 17 00:16:47.925004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.924986    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6x662"
Jul 17 00:16:47.930007 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:47.929981    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8df807_8d49_4d3b_93de_027f715654d0.slice/crio-0ff2c018049bd8ea1258d2160263241f8b88749a26e886a031bd90e5b12386c0 WatchSource:0}: Error finding container 0ff2c018049bd8ea1258d2160263241f8b88749a26e886a031bd90e5b12386c0: Status 404 returned error can't find the container with id 0ff2c018049bd8ea1258d2160263241f8b88749a26e886a031bd90e5b12386c0
Jul 17 00:16:47.932149 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.932134    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8pdf"
Jul 17 00:16:47.935677 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:47.935404    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a2ee30_6f7d_438c_bac7_52add958ae4c.slice/crio-43b0e1ba72d66b6b2580b840acde341b097fcfdfc5abc64cc1982316f02d7d3d WatchSource:0}: Error finding container 43b0e1ba72d66b6b2580b840acde341b097fcfdfc5abc64cc1982316f02d7d3d: Status 404 returned error can't find the container with id 43b0e1ba72d66b6b2580b840acde341b097fcfdfc5abc64cc1982316f02d7d3d
Jul 17 00:16:47.940358 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.940341    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:16:47.948068 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.948050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cdpqd"
Jul 17 00:16:47.950763 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:47.950735    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f892710_cf20_4bdb_aa0b_0c42e8f9436a.slice/crio-abe6d0fbf27d7fd269a018481169b127d3fa698823efb884354f6d0c7e9aa639 WatchSource:0}: Error finding container abe6d0fbf27d7fd269a018481169b127d3fa698823efb884354f6d0c7e9aa639: Status 404 returned error can't find the container with id abe6d0fbf27d7fd269a018481169b127d3fa698823efb884354f6d0c7e9aa639
Jul 17 00:16:47.954988 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:47.954962    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9662c6e8_79c3_4f03_b1bf_e9f85cac2207.slice/crio-2bab45c74a1b9786531f380af72e9c6e6167a1d765838b690b1f5e821ad3a1da WatchSource:0}: Error finding container 2bab45c74a1b9786531f380af72e9c6e6167a1d765838b690b1f5e821ad3a1da: Status 404 returned error can't find the container with id 2bab45c74a1b9786531f380af72e9c6e6167a1d765838b690b1f5e821ad3a1da
Jul 17 00:16:47.966048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.966000    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6x662" event={"ID":"03a2ee30-6f7d-438c-bac7-52add958ae4c","Type":"ContainerStarted","Data":"43b0e1ba72d66b6b2580b840acde341b097fcfdfc5abc64cc1982316f02d7d3d"}
Jul 17 00:16:47.967041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.967018    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7r58z" event={"ID":"bb8df807-8d49-4d3b-93de-027f715654d0","Type":"ContainerStarted","Data":"0ff2c018049bd8ea1258d2160263241f8b88749a26e886a031bd90e5b12386c0"}
Jul 17 00:16:47.967946 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.967926    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" event={"ID":"002d47e5af4c93f026b6cc20e8b1a908","Type":"ContainerStarted","Data":"250c61b58f59ce2e8d72240f37dfd3349599e517ebeb498f312201a98dcafea3"}
Jul 17 00:16:47.968808 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.968770    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"0ea6cd52e772b8df4e90080acb99b082","Type":"ContainerStarted","Data":"858bfafa5f0a29c36fecc0c26abe8399e36d2dd05b3bb460a90ce2da503aa5bb"}
Jul 17 00:16:47.969717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.969696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cdpqd" event={"ID":"9662c6e8-79c3-4f03-b1bf-e9f85cac2207","Type":"ContainerStarted","Data":"2bab45c74a1b9786531f380af72e9c6e6167a1d765838b690b1f5e821ad3a1da"}
Jul 17 00:16:47.970629 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.970597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxwfp" event={"ID":"7f892710-cf20-4bdb-aa0b-0c42e8f9436a","Type":"ContainerStarted","Data":"abe6d0fbf27d7fd269a018481169b127d3fa698823efb884354f6d0c7e9aa639"}
Jul 17 00:16:47.971578 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:47.971560    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerStarted","Data":"38cbef97ea1ac3fec380b28b2ca814dc8a48c89a346103ac718c753559e6b20c"}
Jul 17 00:16:48.003025 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.002991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjnd\" (UniqueName: \"kubernetes.io/projected/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-kube-api-access-nbjnd\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:48.003602 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.003565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:48.003749 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.003644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:48.003749 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.003708    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:48.003865 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.003774    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs podName:7e1bc817-d1b9-458a-a933-8af8ce5e95b3 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:48.503746127 +0000 UTC m=+3.079898682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs") pod "network-metrics-daemon-vjwrp" (UID: "7e1bc817-d1b9-458a-a933-8af8ce5e95b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:48.009468 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.009449    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 00:16:48.009468 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.009469    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 00:16:48.009547 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.009480    2573 projected.go:196] Error preparing data for projected volume kube-api-access-n6z2f for pod openshift-network-diagnostics/network-check-target-7nrcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:48.009581 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.009566    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f podName:51ed8803-1a24-4ff5-8f3f-31b1ccad98ed nodeName:}" failed. No retries permitted until 2026-07-17 00:16:48.50954917 +0000 UTC m=+3.085701733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n6z2f" (UniqueName: "kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f") pod "network-check-target-7nrcr" (UID: "51ed8803-1a24-4ff5-8f3f-31b1ccad98ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:48.011623 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.011596    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjnd\" (UniqueName: \"kubernetes.io/projected/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-kube-api-access-nbjnd\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:48.129020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.128975    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rxcmn"
Jul 17 00:16:48.137089 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:48.137061    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28d955d_d569_4155_8732_247fd315be8f.slice/crio-2886c8f880043a819c93d82897fc2dea0c32db5263df7615c76fe019d3e41d61 WatchSource:0}: Error finding container 2886c8f880043a819c93d82897fc2dea0c32db5263df7615c76fe019d3e41d61: Status 404 returned error can't find the container with id 2886c8f880043a819c93d82897fc2dea0c32db5263df7615c76fe019d3e41d61
Jul 17 00:16:48.186181 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.186142    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6w8v5"
Jul 17 00:16:48.192248 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.192228    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:16:48.193636 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:48.193607    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9896195f_f58d_4c3f_8a1c_c486d1293cc4.slice/crio-e0c0dd9889d58969dbe62f5a7e6bc196796890f21f0af036c31b23b60aee840b WatchSource:0}: Error finding container e0c0dd9889d58969dbe62f5a7e6bc196796890f21f0af036c31b23b60aee840b: Status 404 returned error can't find the container with id e0c0dd9889d58969dbe62f5a7e6bc196796890f21f0af036c31b23b60aee840b
Jul 17 00:16:48.200853 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:48.200828    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983c692a_3a52_4a5d_b36f_4ded907ffd0d.slice/crio-ba87adec4c22d1877d645c8e904d084fa32aac186c04c03479dffe1ad5856192 WatchSource:0}: Error finding container ba87adec4c22d1877d645c8e904d084fa32aac186c04c03479dffe1ad5856192: Status 404 returned error can't find the container with id ba87adec4c22d1877d645c8e904d084fa32aac186c04c03479dffe1ad5856192
Jul 17 00:16:48.210191 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.210167    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx"
Jul 17 00:16:48.213764 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.213739    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 00:16:48.218465 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:16:48.218442    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a18b2f_8553_417d_893b_64d5fc11fc66.slice/crio-2af4eb09339bc621eae6cd061618aea4297dbdabc4f17c633edcaef27f32c51e WatchSource:0}: Error finding container 2af4eb09339bc621eae6cd061618aea4297dbdabc4f17c633edcaef27f32c51e: Status 404 returned error can't find the container with id 2af4eb09339bc621eae6cd061618aea4297dbdabc4f17c633edcaef27f32c51e
Jul 17 00:16:48.225831 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.225810    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 00:16:48.406603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.406566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:48.406826 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.406723    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:48.406826 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.406821    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret podName:fb595638-7449-478a-8505-2e0e531829a1 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:49.406785469 +0000 UTC m=+3.982938026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret") pod "global-pull-secret-syncer-pfsxc" (UID: "fb595638-7449-478a-8505-2e0e531829a1") : object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:48.508032 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.507836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:48.508199 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.508074    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:48.508248 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.508232    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs podName:7e1bc817-d1b9-458a-a933-8af8ce5e95b3 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:49.50820209 +0000 UTC m=+4.084354653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs") pod "network-metrics-daemon-vjwrp" (UID: "7e1bc817-d1b9-458a-a933-8af8ce5e95b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:48.608429 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.608394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:48.608621 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.608597    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 00:16:48.608621 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.608618    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 00:16:48.608733 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.608629    2573 projected.go:196] Error preparing data for projected volume kube-api-access-n6z2f for pod openshift-network-diagnostics/network-check-target-7nrcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:48.608733 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:48.608683    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f podName:51ed8803-1a24-4ff5-8f3f-31b1ccad98ed nodeName:}" failed. No retries permitted until 2026-07-17 00:16:49.60866543 +0000 UTC m=+4.184817996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6z2f" (UniqueName: "kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f") pod "network-check-target-7nrcr" (UID: "51ed8803-1a24-4ff5-8f3f-31b1ccad98ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:48.820768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.820669    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 00:11:47 +0000 UTC" deadline="2028-01-22 01:51:46.043674763 +0000 UTC"
Jul 17 00:16:48.820768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.820726    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13297h34m57.222952695s"
Jul 17 00:16:48.976152 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.976118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx" event={"ID":"88a18b2f-8553-417d-893b-64d5fc11fc66","Type":"ContainerStarted","Data":"2af4eb09339bc621eae6cd061618aea4297dbdabc4f17c633edcaef27f32c51e"}
Jul 17 00:16:48.978337 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.978242    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"ba87adec4c22d1877d645c8e904d084fa32aac186c04c03479dffe1ad5856192"}
Jul 17 00:16:48.980361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.980285    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6w8v5" event={"ID":"9896195f-f58d-4c3f-8a1c-c486d1293cc4","Type":"ContainerStarted","Data":"e0c0dd9889d58969dbe62f5a7e6bc196796890f21f0af036c31b23b60aee840b"}
Jul 17 00:16:48.981717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:48.981693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rxcmn" event={"ID":"d28d955d-d569-4155-8732-247fd315be8f","Type":"ContainerStarted","Data":"2886c8f880043a819c93d82897fc2dea0c32db5263df7615c76fe019d3e41d61"}
Jul 17 00:16:49.414418 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.414381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:49.414613 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.414597    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:49.414701 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.414689    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret podName:fb595638-7449-478a-8505-2e0e531829a1 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:51.414670729 +0000 UTC m=+5.990823283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret") pod "global-pull-secret-syncer-pfsxc" (UID: "fb595638-7449-478a-8505-2e0e531829a1") : object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:49.515275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.515243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:49.515409 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.515387    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:49.515481 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.515446    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs podName:7e1bc817-d1b9-458a-a933-8af8ce5e95b3 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:51.515428936 +0000 UTC m=+6.091581498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs") pod "network-metrics-daemon-vjwrp" (UID: "7e1bc817-d1b9-458a-a933-8af8ce5e95b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:49.615973 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.615926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:49.616126 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.616079    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 00:16:49.616126 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.616099    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 00:16:49.616126 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.616111    2573 projected.go:196] Error preparing data for projected volume kube-api-access-n6z2f for pod openshift-network-diagnostics/network-check-target-7nrcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:49.616416 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.616164    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f podName:51ed8803-1a24-4ff5-8f3f-31b1ccad98ed nodeName:}" failed. No retries permitted until 2026-07-17 00:16:51.61614755 +0000 UTC m=+6.192300119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6z2f" (UniqueName: "kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f") pod "network-check-target-7nrcr" (UID: "51ed8803-1a24-4ff5-8f3f-31b1ccad98ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:49.821497 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.821459    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 00:11:47 +0000 UTC" deadline="2028-01-29 07:28:49.411357585 +0000 UTC"
Jul 17 00:16:49.821497 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.821496    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13471h11m59.589866082s"
Jul 17 00:16:49.964320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.963890    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:49.964320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.963906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:49.964320 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.964007    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:16:49.964320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:49.964057    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:49.964320 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.964139    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:16:49.964320 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:49.964224    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:16:51.432104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:51.432069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:51.432554 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.432242    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:51.432554 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.432313    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret podName:fb595638-7449-478a-8505-2e0e531829a1 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:55.432293387 +0000 UTC m=+10.008445952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret") pod "global-pull-secret-syncer-pfsxc" (UID: "fb595638-7449-478a-8505-2e0e531829a1") : object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:51.532651 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:51.532614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:51.532870 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.532851    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:51.532969 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.532920    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs podName:7e1bc817-d1b9-458a-a933-8af8ce5e95b3 nodeName:}" failed. No retries permitted until 2026-07-17 00:16:55.532901236 +0000 UTC m=+10.109053811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs") pod "network-metrics-daemon-vjwrp" (UID: "7e1bc817-d1b9-458a-a933-8af8ce5e95b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:51.633733 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:51.633196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:51.633733 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.633339    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 00:16:51.633733 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.633359    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 00:16:51.633733 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.633372    2573 projected.go:196] Error preparing data for projected volume kube-api-access-n6z2f for pod openshift-network-diagnostics/network-check-target-7nrcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:51.633733 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.633427    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f podName:51ed8803-1a24-4ff5-8f3f-31b1ccad98ed nodeName:}" failed. No retries permitted until 2026-07-17 00:16:55.633409748 +0000 UTC m=+10.209562316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6z2f" (UniqueName: "kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f") pod "network-check-target-7nrcr" (UID: "51ed8803-1a24-4ff5-8f3f-31b1ccad98ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:51.975916 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:51.974915    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:51.975916 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.975096    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:16:51.975916 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:51.975551    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:51.975916 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.975670    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:16:51.975916 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:51.975746    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:51.975916 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:51.975863    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:16:53.962806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:53.962552    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:53.962806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:53.962598    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:53.962806 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:53.962689    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:16:53.962806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:53.962775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:53.963400 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:53.962873    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:16:53.963400 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:53.962950    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:16:55.462508 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:55.462473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:55.462990 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.462664    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:55.462990 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.462721    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret podName:fb595638-7449-478a-8505-2e0e531829a1 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:03.462702697 +0000 UTC m=+18.038855254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret") pod "global-pull-secret-syncer-pfsxc" (UID: "fb595638-7449-478a-8505-2e0e531829a1") : object "kube-system"/"original-pull-secret" not registered
Jul 17 00:16:55.563360 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:55.563320    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:55.563516 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.563474    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:55.563561 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.563541    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs podName:7e1bc817-d1b9-458a-a933-8af8ce5e95b3 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:03.563522149 +0000 UTC m=+18.139674725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs") pod "network-metrics-daemon-vjwrp" (UID: "7e1bc817-d1b9-458a-a933-8af8ce5e95b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:16:55.664342 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:55.664084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:55.664342 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.664264    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 00:16:55.664342 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.664289    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 00:16:55.664342 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.664301    2573 projected.go:196] Error preparing data for projected volume kube-api-access-n6z2f for pod openshift-network-diagnostics/network-check-target-7nrcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:55.664637 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.664362    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f podName:51ed8803-1a24-4ff5-8f3f-31b1ccad98ed nodeName:}" failed. No retries permitted until 2026-07-17 00:17:03.664345307 +0000 UTC m=+18.240497860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6z2f" (UniqueName: "kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f") pod "network-check-target-7nrcr" (UID: "51ed8803-1a24-4ff5-8f3f-31b1ccad98ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:16:55.962968 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:55.962935    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:55.963803 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.963046    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:16:55.963956 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:55.963864    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:55.964028 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.964009    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:16:55.964078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:55.964069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:55.964198 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:55.964178    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:16:57.962596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:57.962569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:57.963023 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:57.962617    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:57.963023 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:57.962573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:57.963023 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:57.962705    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:16:57.963023 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:57.962770    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:16:57.963023 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:57.962851    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:16:59.962519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:59.962480    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:16:59.963024 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:59.962477    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:16:59.963024 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:59.962604    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:16:59.963024 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:16:59.962619    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:16:59.963024 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:59.962703    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:16:59.963024 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:16:59.962754    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:01.962580 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:01.962540    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:01.962580 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:01.962572    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:01.963094 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:01.962545    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:01.963094 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:01.962702    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:01.963094 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:01.962844    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:01.963094 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:01.962926    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:03.523930 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:03.523895    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:03.524318 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.524038    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 00:17:03.524318 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.524095    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret podName:fb595638-7449-478a-8505-2e0e531829a1 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:19.52408093 +0000 UTC m=+34.100233482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret") pod "global-pull-secret-syncer-pfsxc" (UID: "fb595638-7449-478a-8505-2e0e531829a1") : object "kube-system"/"original-pull-secret" not registered
Jul 17 00:17:03.624684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:03.624650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:03.624877 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.624829    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:17:03.624940 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.624898    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs podName:7e1bc817-d1b9-458a-a933-8af8ce5e95b3 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:19.624878716 +0000 UTC m=+34.201031268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs") pod "network-metrics-daemon-vjwrp" (UID: "7e1bc817-d1b9-458a-a933-8af8ce5e95b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:17:03.725611 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:03.725577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:03.725779 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.725745    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 00:17:03.725779 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.725762    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 00:17:03.725779 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.725772    2573 projected.go:196] Error preparing data for projected volume kube-api-access-n6z2f for pod openshift-network-diagnostics/network-check-target-7nrcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:17:03.725961 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.725847    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f podName:51ed8803-1a24-4ff5-8f3f-31b1ccad98ed nodeName:}" failed. No retries permitted until 2026-07-17 00:17:19.725826594 +0000 UTC m=+34.301979147 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6z2f" (UniqueName: "kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f") pod "network-check-target-7nrcr" (UID: "51ed8803-1a24-4ff5-8f3f-31b1ccad98ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:17:03.962983 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:03.962850    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:03.962983 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:03.962870    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:03.962983 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.962970    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:03.963224 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.963014    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:03.963224 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:03.963034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:03.963224 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:03.963151    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:05.962580 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:05.962545    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:05.963031 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:05.962638    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:05.963031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:05.962991    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:05.963157 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:05.963082    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:05.963157 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:05.963145    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:05.963297 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:05.963274    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:07.966390 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:07.966202    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:07.966767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:07.966283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:07.966767 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:07.966478    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:07.966767 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:07.966575    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:07.966767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:07.966294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:07.966767 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:07.966673    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:08.023756 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:08.023733    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" event={"ID":"002d47e5af4c93f026b6cc20e8b1a908","Type":"ContainerStarted","Data":"5750e195caba39ad4b99e2e9d2c14725fd458814b811ecdcbf1e7c5e859807df"}
Jul 17 00:17:08.025176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:08.025152    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cdpqd" event={"ID":"9662c6e8-79c3-4f03-b1bf-e9f85cac2207","Type":"ContainerStarted","Data":"622b920ead866c7c181a7059eef0bddf31eee9dd83d5aa2441d17340d142c869"}
Jul 17 00:17:08.027319 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:08.027297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"7b4742df78165577aadfe45183ab7c235341524dce3d729cd26b9cae07051207"}
Jul 17 00:17:08.028580 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:08.028563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7r58z" event={"ID":"bb8df807-8d49-4d3b-93de-027f715654d0","Type":"ContainerStarted","Data":"c1e4ed7863037192afb13a865382f3ab1416e4d0097e81f56d239c589ecc02de"}
Jul 17 00:17:08.038659 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:08.038497    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" podStartSLOduration=22.038483417 podStartE2EDuration="22.038483417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:08.038425043 +0000 UTC m=+22.614577618" watchObservedRunningTime="2026-07-17 00:17:08.038483417 +0000 UTC m=+22.614635992"
Jul 17 00:17:08.087065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:08.087029    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7r58z" podStartSLOduration=3.882540665 podStartE2EDuration="23.087017427s" totalImagesPullingTime="19.204476762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:08.086900761 +0000 UTC m=+22.663053347" watchObservedRunningTime="2026-07-17 00:17:08.087017427 +0000 UTC m=+22.663170001"
Jul 17 00:17:08.087264 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:08.087243    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cdpqd" podStartSLOduration=3.638024961 podStartE2EDuration="23.087237028s" totalImagesPullingTime="19.449212067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:08.062489635 +0000 UTC m=+22.638642210" watchObservedRunningTime="2026-07-17 00:17:08.087237028 +0000 UTC m=+22.663389603"
Jul 17 00:17:09.033093 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.032814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxwfp" event={"ID":"7f892710-cf20-4bdb-aa0b-0c42e8f9436a","Type":"ContainerStarted","Data":"a133417dfe1280bdfb1538d492ccd0fbf61333ac353906f3b96375bf60aef511"}
Jul 17 00:17:09.035091 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.035063    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ea6cd52e772b8df4e90080acb99b082" containerID="0ece535a5e4edff83016b857dc13bb6e1657e113a3fd447fe5a8eca341bfefdb" exitCode=0
Jul 17 00:17:09.035231 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.035141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"0ea6cd52e772b8df4e90080acb99b082","Type":"ContainerDied","Data":"0ece535a5e4edff83016b857dc13bb6e1657e113a3fd447fe5a8eca341bfefdb"}
Jul 17 00:17:09.036554 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.036532    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx" event={"ID":"88a18b2f-8553-417d-893b-64d5fc11fc66","Type":"ContainerStarted","Data":"ba4f480a71ae0391a0c88d87b1f9d318a473ea6b0ba01eae84078f3e62d490f8"}
Jul 17 00:17:09.039244 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.039227    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:17:09.039578 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.039559    2573 generic.go:356] "Generic (PLEG): container finished" podID="983c692a-3a52-4a5d-b36f-4ded907ffd0d" containerID="3723922d74ce03a248d11913c9f8d3dd6f2bb799f15a7df85932f9b7107ed106" exitCode=1
Jul 17 00:17:09.039648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.039625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerDied","Data":"3723922d74ce03a248d11913c9f8d3dd6f2bb799f15a7df85932f9b7107ed106"}
Jul 17 00:17:09.039695 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.039684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"5f34175a76afa966bc16d657d34d7a62d7a1b6637a6ea097b89a6c3616471c79"}
Jul 17 00:17:09.039745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.039714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"dcbbccdc11af61362c7d9c6ba1524c81310ae306d048da7065319668d1fa3710"}
Jul 17 00:17:09.039801 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.039746    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"acec38387bb50292eaa31e7f2de87b8ee6105b0778fdda293e39c2c331b78532"}
Jul 17 00:17:09.039801 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.039774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"f7a262ff853c23af00b33a02cee709504543329295d759120c3e16c36a1f2ff3"}
Jul 17 00:17:09.040927 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.040910    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6w8v5" event={"ID":"9896195f-f58d-4c3f-8a1c-c486d1293cc4","Type":"ContainerStarted","Data":"4b62f96eaa6718e4b65fa8f684c4458716417688dbd11abc09ad9e5060b7307d"}
Jul 17 00:17:09.042263 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.042243    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rxcmn" event={"ID":"d28d955d-d569-4155-8732-247fd315be8f","Type":"ContainerStarted","Data":"56d78f6fbac23c661432a1595d70d900b3f7e6871f6e3b42b1c87db07676f02d"}
Jul 17 00:17:09.043638 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.043618    2573 generic.go:356] "Generic (PLEG): container finished" podID="00c347f5-1f2b-402c-8edf-cc04d4832887" containerID="8b747ca4e093640335f0f34848f695387df7012f257dc0802e675898e9b1d665" exitCode=0
Jul 17 00:17:09.043722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.043690    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerDied","Data":"8b747ca4e093640335f0f34848f695387df7012f257dc0802e675898e9b1d665"}
Jul 17 00:17:09.044975 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.044956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6x662" event={"ID":"03a2ee30-6f7d-438c-bac7-52add958ae4c","Type":"ContainerStarted","Data":"e4c8a2d4deaccd4ab29b61c48f67f04f5b977703953c723ab804867bf2317bec"}
Jul 17 00:17:09.045406 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.045366    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zxwfp" podStartSLOduration=4.837799663 podStartE2EDuration="24.045353406s" totalImagesPullingTime="19.207553743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:09.044673873 +0000 UTC m=+23.620826447" watchObservedRunningTime="2026-07-17 00:17:09.045353406 +0000 UTC m=+23.621505980"
Jul 17 00:17:09.086373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.086188    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rxcmn" podStartSLOduration=5.064674229 podStartE2EDuration="24.086174949s" totalImagesPullingTime="19.02150072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:09.085806454 +0000 UTC m=+23.661959030" watchObservedRunningTime="2026-07-17 00:17:09.086174949 +0000 UTC m=+23.662327522"
Jul 17 00:17:09.097192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.097155    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6w8v5" podStartSLOduration=5.130578947 podStartE2EDuration="24.09714575s" totalImagesPullingTime="18.966566803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:09.096979896 +0000 UTC m=+23.673132481" watchObservedRunningTime="2026-07-17 00:17:09.09714575 +0000 UTC m=+23.673298323"
Jul 17 00:17:09.111633 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.111590    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6x662" podStartSLOduration=13.349019085 podStartE2EDuration="24.111582345s" totalImagesPullingTime="10.76256326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:09.111580628 +0000 UTC m=+23.687733202" watchObservedRunningTime="2026-07-17 00:17:09.111582345 +0000 UTC m=+23.687734919"
Jul 17 00:17:09.587746 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.587720    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 00:17:09.858456 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.858351    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T00:17:09.587742916Z","UUID":"8ca72855-6308-4441-afc3-38cda716b34e","Handler":null,"Name":"","Endpoint":""}
Jul 17 00:17:09.859830 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.859803    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 00:17:09.859947 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.859844    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 00:17:09.962906 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.962868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:09.963157 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:09.962970    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:09.963157 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.962868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:09.963157 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:09.963007    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:09.963157 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:09.963082    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:09.963374 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:09.963160    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:10.049444 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:10.049410    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx" event={"ID":"88a18b2f-8553-417d-893b-64d5fc11fc66","Type":"ContainerStarted","Data":"47ce6d53c21a1767c5ac53056a0fae60a93d663fe850d257ff7d0341c39eb9d5"}
Jul 17 00:17:10.051348 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:10.051296    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"0ea6cd52e772b8df4e90080acb99b082","Type":"ContainerStarted","Data":"6490f795f929ff0f48b26a14378f9df923ea6d3d7f8d74fb421ecc4a7aa7049b"}
Jul 17 00:17:10.064985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:10.064937    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" podStartSLOduration=24.064920547 podStartE2EDuration="24.064920547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:10.064690108 +0000 UTC m=+24.640842683" watchObservedRunningTime="2026-07-17 00:17:10.064920547 +0000 UTC m=+24.641073120"
Jul 17 00:17:11.056167 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.056125    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx" event={"ID":"88a18b2f-8553-417d-893b-64d5fc11fc66","Type":"ContainerStarted","Data":"ff9bc7cd2589424b81f92b19b78609e9b0edb5043648e38ae3f93af74c786aa3"}
Jul 17 00:17:11.059723 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.059702    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:17:11.060152 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.060131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"95aef98ae2a5c4ffe95291c5cc59b86c10e5c351beeda1c00e17f177f14c8748"}
Jul 17 00:17:11.075519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.075484    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:17:11.076052 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.076035    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:17:11.097885 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.097763    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zd5vx" podStartSLOduration=4.764337298 podStartE2EDuration="26.097703322s" totalImagesPullingTime="21.333366024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:11.09750068 +0000 UTC m=+25.673653254" watchObservedRunningTime="2026-07-17 00:17:11.097703322 +0000 UTC m=+25.673855895"
Jul 17 00:17:11.962183 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.962150    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:11.962361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.962150    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:11.962361 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:11.962264    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:11.962484 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:11.962152    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:11.962484 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:11.962472    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:11.962587 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:11.962349    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:12.062558 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:12.062524    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:17:12.063476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:12.063456    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zxwfp"
Jul 17 00:17:13.773231 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:13.773205    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6x662_03a2ee30-6f7d-438c-bac7-52add958ae4c/dns-node-resolver/0.log"
Jul 17 00:17:13.962706 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:13.962675    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:13.962883 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:13.962718    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:13.962883 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:13.962675    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:13.962883 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:13.962825    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:13.963044 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:13.962892    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:13.963044 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:13.962952    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:14.947287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:14.947124    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rxcmn_d28d955d-d569-4155-8732-247fd315be8f/node-ca/0.log"
Jul 17 00:17:15.068896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.068865    2573 generic.go:356] "Generic (PLEG): container finished" podID="00c347f5-1f2b-402c-8edf-cc04d4832887" containerID="f6494c665f7d182bf37527250fe1a72e9661bd4337423e82faa978497921c305" exitCode=0
Jul 17 00:17:15.069068 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.068950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerDied","Data":"f6494c665f7d182bf37527250fe1a72e9661bd4337423e82faa978497921c305"}
Jul 17 00:17:15.072455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.072434    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:17:15.072890 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.072849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"f09aa44a4083b7a506fff9f88a084a9ffc9119cc325019879fdad8c3514b11b4"}
Jul 17 00:17:15.073131 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.073111    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:17:15.073271 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.073257    2573 scope.go:117] "RemoveContainer" containerID="3723922d74ce03a248d11913c9f8d3dd6f2bb799f15a7df85932f9b7107ed106"
Jul 17 00:17:15.089208 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.089193    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:17:15.533360 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.533312    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:17:15.963367 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.963060    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:15.963868 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.963135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:15.963868 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:15.963399    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:15.963868 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:15.963175    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:15.963868 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:15.963501    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:15.963868 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:15.963577    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:16.081078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.081051    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:17:16.081529 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.081502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" event={"ID":"983c692a-3a52-4a5d-b36f-4ded907ffd0d","Type":"ContainerStarted","Data":"d4dfc43755623163d4579c2d6e85e6f613dbdf09be906efec79a9f4818bd0b4c"}
Jul 17 00:17:16.081783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.081762    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:17:16.097386 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.097364    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:17:16.111081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.111038    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-645g2" podStartSLOduration=11.486281263 podStartE2EDuration="31.111019813s" totalImagesPullingTime="19.62473855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:16.110435814 +0000 UTC m=+30.686588387" watchObservedRunningTime="2026-07-17 00:17:16.111019813 +0000 UTC m=+30.687172370"
Jul 17 00:17:16.514856 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.514806    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pfsxc"]
Jul 17 00:17:16.515062 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.514975    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:16.515133 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:16.515103    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:16.515528 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.515506    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vjwrp"]
Jul 17 00:17:16.515624 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.515597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:16.515693 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:16.515674    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:16.516177 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.516153    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7nrcr"]
Jul 17 00:17:16.516296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:16.516229    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:16.516352 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:16.516305    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:17.085389 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:17.085301    2573 generic.go:356] "Generic (PLEG): container finished" podID="00c347f5-1f2b-402c-8edf-cc04d4832887" containerID="989f879142ac5b8a0f1811cf0356ea999d6ec76bb935b612629b70288056ed4e" exitCode=0
Jul 17 00:17:17.085745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:17.085384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerDied","Data":"989f879142ac5b8a0f1811cf0356ea999d6ec76bb935b612629b70288056ed4e"}
Jul 17 00:17:17.962342 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:17.962321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:17.962459 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:17.962322    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:17.962459 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:17.962432    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:17.962566 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:17.962543    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:18.089368 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:18.089339    2573 generic.go:356] "Generic (PLEG): container finished" podID="00c347f5-1f2b-402c-8edf-cc04d4832887" containerID="14f58c51a0622dca3a045d64dfc503b1e201bb0406e945c71d1039fa1311d1dd" exitCode=0
Jul 17 00:17:18.089841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:18.089429    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerDied","Data":"14f58c51a0622dca3a045d64dfc503b1e201bb0406e945c71d1039fa1311d1dd"}
Jul 17 00:17:18.962872 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:18.962843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:18.963014 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:18.962945    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:19.547881 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:19.547842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:19.548268 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.547970    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 00:17:19.548268 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.548029    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret podName:fb595638-7449-478a-8505-2e0e531829a1 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:51.548013648 +0000 UTC m=+66.124166218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret") pod "global-pull-secret-syncer-pfsxc" (UID: "fb595638-7449-478a-8505-2e0e531829a1") : object "kube-system"/"original-pull-secret" not registered
Jul 17 00:17:19.649143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:19.649107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:19.649311 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.649210    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:17:19.649311 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.649260    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs podName:7e1bc817-d1b9-458a-a933-8af8ce5e95b3 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:51.649247504 +0000 UTC m=+66.225400056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs") pod "network-metrics-daemon-vjwrp" (UID: "7e1bc817-d1b9-458a-a933-8af8ce5e95b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 00:17:19.750435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:19.750402    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:19.750644 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.750598    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 00:17:19.750644 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.750623    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 00:17:19.750644 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.750636    2573 projected.go:196] Error preparing data for projected volume kube-api-access-n6z2f for pod openshift-network-diagnostics/network-check-target-7nrcr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:17:19.750824 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.750703    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f podName:51ed8803-1a24-4ff5-8f3f-31b1ccad98ed nodeName:}" failed. No retries permitted until 2026-07-17 00:17:51.750684085 +0000 UTC m=+66.326836654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6z2f" (UniqueName: "kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f") pod "network-check-target-7nrcr" (UID: "51ed8803-1a24-4ff5-8f3f-31b1ccad98ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 00:17:19.962814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:19.962717    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:19.962982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:19.962717    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:19.962982 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.962884    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:19.962982 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:19.962924    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:20.962020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:20.961984    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:20.962422 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:20.962095    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:21.962287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:21.962253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:21.962771 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:21.962253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:21.962771 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:21.962382    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:21.962771 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:21.962488    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:22.962298 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:22.962265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:22.962467 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:22.962377    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:23.962567 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:23.962354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:23.963048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:23.962418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:23.963048 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:23.962683    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:23.963048 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:23.962756    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:24.962072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:24.962039    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:24.962230 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:24.962148    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:25.963702 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:25.963653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:25.964306 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:25.963785    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:25.964306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:25.963825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:25.964306 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:25.963908    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:26.109477 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:26.109383    2573 generic.go:356] "Generic (PLEG): container finished" podID="00c347f5-1f2b-402c-8edf-cc04d4832887" containerID="cc31635bbd0c85badb4165c842f3dd59bb49d2b6a6581eeecf6cc5a507e9d900" exitCode=0
Jul 17 00:17:26.109477 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:26.109433    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerDied","Data":"cc31635bbd0c85badb4165c842f3dd59bb49d2b6a6581eeecf6cc5a507e9d900"}
Jul 17 00:17:26.962244 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:26.962200    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:26.962408 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:26.962313    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:27.114694 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:27.114653    2573 generic.go:356] "Generic (PLEG): container finished" podID="00c347f5-1f2b-402c-8edf-cc04d4832887" containerID="e83094ca303577c4dacbe82a8e8dcc87a81dbf525f325a77dc284eb658e0cd5a" exitCode=0
Jul 17 00:17:27.114694 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:27.114694    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerDied","Data":"e83094ca303577c4dacbe82a8e8dcc87a81dbf525f325a77dc284eb658e0cd5a"}
Jul 17 00:17:27.962756 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:27.962728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:27.962973 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:27.962853    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:27.962973 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:27.962935    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:27.963050 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:27.963020    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:28.120283 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:28.120245    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" event={"ID":"00c347f5-1f2b-402c-8edf-cc04d4832887","Type":"ContainerStarted","Data":"1d3f190c5331c94d19e04f6cd0018b8d33e02ce689e4fc44b2a4824d3e5139e7"}
Jul 17 00:17:28.141264 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:28.141202    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k8pdf" podStartSLOduration=9.811931534 podStartE2EDuration="43.141180058s" totalImagesPullingTime="33.329248524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:28.140563778 +0000 UTC m=+42.716716353" watchObservedRunningTime="2026-07-17 00:17:28.141180058 +0000 UTC m=+42.717332636"
Jul 17 00:17:28.962464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:28.962418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:28.962623 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:28.962548    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:29.962274 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:29.962237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:29.962695 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:29.962363    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:29.962695 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:29.962395    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:29.962695 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:29.962445    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:30.962313 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:30.962277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:30.962716 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:30.962389    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:31.962694 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:31.962658    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:31.963173 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:31.962663    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:31.963173 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:31.962784    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:31.963173 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:31.962853    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:32.962353 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:32.962313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:32.962541 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:32.962421    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:33.962864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:33.962829    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:33.962864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:33.962850    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:33.963399 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:33.962965    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:33.963399 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:33.963072    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:34.962353 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:34.962311    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:34.962548 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:34.962437    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfsxc" podUID="fb595638-7449-478a-8505-2e0e531829a1"
Jul 17 00:17:35.964817 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:35.964766    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:35.965329 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:35.964961    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjwrp" podUID="7e1bc817-d1b9-458a-a933-8af8ce5e95b3"
Jul 17 00:17:35.965329 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:35.965253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:35.965449 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:35.965368    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nrcr" podUID="51ed8803-1a24-4ff5-8f3f-31b1ccad98ed"
Jul 17 00:17:36.286625 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.286598    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeReady"
Jul 17 00:17:36.286820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.286725    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 00:17:36.319196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.319151    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c97cc46cb-jmhcr"]
Jul 17 00:17:36.331615 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.331587    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xkcbl"]
Jul 17 00:17:36.331774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.331724    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.333451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.333426    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 00:17:36.333589 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.333529    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 00:17:36.333743 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.333729    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kvnl8\""
Jul 17 00:17:36.333844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.333829    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 00:17:36.338882 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.338861    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 00:17:36.345674 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.345649    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zxczp"]
Jul 17 00:17:36.345784 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.345774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.348262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.347998    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mrnf5\""
Jul 17 00:17:36.348262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.348084    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 00:17:36.348262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.348119    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 00:17:36.348488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.348297    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 00:17:36.348488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.348392    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 00:17:36.358311 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-trusted-ca\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.358443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358316    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b5h\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-kube-api-access-99b5h\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.358443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358350    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78bc1249-e4fb-49bd-8d71-ba85d657c82d-crio-socket\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.358443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78bc1249-e4fb-49bd-8d71-ba85d657c82d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.358443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78bc1249-e4fb-49bd-8d71-ba85d657c82d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.358605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358447    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-image-registry-private-configuration\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.358605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-ca-trust-extracted\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.358605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358499    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-bound-sa-token\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.358605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmczh\" (UniqueName: \"kubernetes.io/projected/78bc1249-e4fb-49bd-8d71-ba85d657c82d-kube-api-access-jmczh\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.358605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-registry-tls\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.358605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358580    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-installation-pull-secrets\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.358819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358616    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78bc1249-e4fb-49bd-8d71-ba85d657c82d-data-volume\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.358819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.358632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-registry-certificates\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.365171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.365153    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c97cc46cb-jmhcr"]
Jul 17 00:17:36.365241 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.365183    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xkcbl"]
Jul 17 00:17:36.365241 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.365204    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zxczp"]
Jul 17 00:17:36.365327 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.365294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.367220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.367192    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9rpn\""
Jul 17 00:17:36.367347 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.367248    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 00:17:36.367347 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.367269    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 00:17:36.435005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.434970    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ph4x4"]
Jul 17 00:17:36.452576 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.452549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ph4x4"]
Jul 17 00:17:36.452740 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.452646    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.454891 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.454864    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mxbsx\""
Jul 17 00:17:36.454891 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.454885    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 00:17:36.455098 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.454889    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 00:17:36.455098 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.454968    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 00:17:36.459071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.458906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgsmq\" (UniqueName: \"kubernetes.io/projected/74ed1822-52df-4463-87a9-cd5367dfca3a-kube-api-access-cgsmq\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.459071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.458976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmczh\" (UniqueName: \"kubernetes.io/projected/78bc1249-e4fb-49bd-8d71-ba85d657c82d-kube-api-access-jmczh\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.459071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-registry-tls\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.459300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ed1822-52df-4463-87a9-cd5367dfca3a-tmp-dir\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.459300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-installation-pull-secrets\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.459300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78bc1249-e4fb-49bd-8d71-ba85d657c82d-data-volume\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.459300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-registry-certificates\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.459506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459305    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-trusted-ca\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.459506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b5h\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-kube-api-access-99b5h\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.459506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74ed1822-52df-4463-87a9-cd5367dfca3a-config-volume\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.459506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74ed1822-52df-4463-87a9-cd5367dfca3a-metrics-tls\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.459506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78bc1249-e4fb-49bd-8d71-ba85d657c82d-crio-socket\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.459506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78bc1249-e4fb-49bd-8d71-ba85d657c82d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.459506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459492    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78bc1249-e4fb-49bd-8d71-ba85d657c82d-data-volume\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.459925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78bc1249-e4fb-49bd-8d71-ba85d657c82d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.459925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-image-registry-private-configuration\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.459925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-ca-trust-extracted\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.459925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.459603    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-bound-sa-token\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.460122 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.460031    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78bc1249-e4fb-49bd-8d71-ba85d657c82d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.460176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.460150    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-registry-certificates\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.460237 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.460194    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78bc1249-e4fb-49bd-8d71-ba85d657c82d-crio-socket\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.460424 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.460387    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-trusted-ca\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.460534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.460501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-ca-trust-extracted\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.463583 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.463562    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-image-registry-private-configuration\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.463698 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.463583    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78bc1249-e4fb-49bd-8d71-ba85d657c82d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.463698 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.463664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-installation-pull-secrets\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.463935 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.463919    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-registry-tls\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.468820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.468775    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-bound-sa-token\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.469009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.468991    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b5h\" (UniqueName: \"kubernetes.io/projected/8f9d1891-b7c3-42a4-951b-4145be0e8ffb-kube-api-access-99b5h\") pod \"image-registry-6c97cc46cb-jmhcr\" (UID: \"8f9d1891-b7c3-42a4-951b-4145be0e8ffb\") " pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.479026 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.478993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmczh\" (UniqueName: \"kubernetes.io/projected/78bc1249-e4fb-49bd-8d71-ba85d657c82d-kube-api-access-jmczh\") pod \"insights-runtime-extractor-xkcbl\" (UID: \"78bc1249-e4fb-49bd-8d71-ba85d657c82d\") " pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.561018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.560921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74ed1822-52df-4463-87a9-cd5367dfca3a-config-volume\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.561018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.560964    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74ed1822-52df-4463-87a9-cd5367dfca3a-metrics-tls\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.561018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.560988    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/863459a2-790d-4fab-afd5-b9a255285b3a-cert\") pod \"ingress-canary-ph4x4\" (UID: \"863459a2-790d-4fab-afd5-b9a255285b3a\") " pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.561287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.561034    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgsmq\" (UniqueName: \"kubernetes.io/projected/74ed1822-52df-4463-87a9-cd5367dfca3a-kube-api-access-cgsmq\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.561287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.561069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bnv\" (UniqueName: \"kubernetes.io/projected/863459a2-790d-4fab-afd5-b9a255285b3a-kube-api-access-s2bnv\") pod \"ingress-canary-ph4x4\" (UID: \"863459a2-790d-4fab-afd5-b9a255285b3a\") " pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.561287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.561092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ed1822-52df-4463-87a9-cd5367dfca3a-tmp-dir\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.561485 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.561450    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ed1822-52df-4463-87a9-cd5367dfca3a-tmp-dir\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.561641 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.561618    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74ed1822-52df-4463-87a9-cd5367dfca3a-config-volume\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.563473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.563452    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74ed1822-52df-4463-87a9-cd5367dfca3a-metrics-tls\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.569405 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.569380    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgsmq\" (UniqueName: \"kubernetes.io/projected/74ed1822-52df-4463-87a9-cd5367dfca3a-kube-api-access-cgsmq\") pod \"dns-default-zxczp\" (UID: \"74ed1822-52df-4463-87a9-cd5367dfca3a\") " pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.642401 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.642368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:36.655685 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.655651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xkcbl"
Jul 17 00:17:36.661907 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.661881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/863459a2-790d-4fab-afd5-b9a255285b3a-cert\") pod \"ingress-canary-ph4x4\" (UID: \"863459a2-790d-4fab-afd5-b9a255285b3a\") " pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.662023 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.661956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bnv\" (UniqueName: \"kubernetes.io/projected/863459a2-790d-4fab-afd5-b9a255285b3a-kube-api-access-s2bnv\") pod \"ingress-canary-ph4x4\" (UID: \"863459a2-790d-4fab-afd5-b9a255285b3a\") " pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.664389 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.664369    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/863459a2-790d-4fab-afd5-b9a255285b3a-cert\") pod \"ingress-canary-ph4x4\" (UID: \"863459a2-790d-4fab-afd5-b9a255285b3a\") " pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.669560 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.669529    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bnv\" (UniqueName: \"kubernetes.io/projected/863459a2-790d-4fab-afd5-b9a255285b3a-kube-api-access-s2bnv\") pod \"ingress-canary-ph4x4\" (UID: \"863459a2-790d-4fab-afd5-b9a255285b3a\") " pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.674335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.674318    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:36.762589 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.762554    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ph4x4"
Jul 17 00:17:36.903526 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.903483    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c97cc46cb-jmhcr"]
Jul 17 00:17:36.915770 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.915743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zxczp"]
Jul 17 00:17:36.919374 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:36.919340    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ed1822_52df_4463_87a9_cd5367dfca3a.slice/crio-491bc2001b11ae076ea66e0aed00dd2b6b4fb1ebc634707448a2e40717508c2c WatchSource:0}: Error finding container 491bc2001b11ae076ea66e0aed00dd2b6b4fb1ebc634707448a2e40717508c2c: Status 404 returned error can't find the container with id 491bc2001b11ae076ea66e0aed00dd2b6b4fb1ebc634707448a2e40717508c2c
Jul 17 00:17:36.925331 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.925298    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xkcbl"]
Jul 17 00:17:36.929210 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:36.929180    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bc1249_e4fb_49bd_8d71_ba85d657c82d.slice/crio-d83d698392f73aad70666b5049ccb438343373a516832c2cf730d306f40bb20e WatchSource:0}: Error finding container d83d698392f73aad70666b5049ccb438343373a516832c2cf730d306f40bb20e: Status 404 returned error can't find the container with id d83d698392f73aad70666b5049ccb438343373a516832c2cf730d306f40bb20e
Jul 17 00:17:36.948634 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.948606    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-skjmc"]
Jul 17 00:17:36.976134 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:36.976101    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863459a2_790d_4fab_afd5_b9a255285b3a.slice/crio-90188bd8ce658d08cc3330e2ba9f60c46c76671ce63142d5b1a60e7d4e1b2aa4 WatchSource:0}: Error finding container 90188bd8ce658d08cc3330e2ba9f60c46c76671ce63142d5b1a60e7d4e1b2aa4: Status 404 returned error can't find the container with id 90188bd8ce658d08cc3330e2ba9f60c46c76671ce63142d5b1a60e7d4e1b2aa4
Jul 17 00:17:36.982094 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.982063    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-skjmc"]
Jul 17 00:17:36.982189 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.982102    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ph4x4"]
Jul 17 00:17:36.982228 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.982219    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:36.982260 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.982228    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-skjmc"
Jul 17 00:17:36.984157 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.984132    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 00:17:36.984269 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.984212    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 00:17:36.984395 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.984378    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bkktw\""
Jul 17 00:17:36.984438 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:36.984419    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 00:17:37.065626 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.065596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54fmr\" (UniqueName: \"kubernetes.io/projected/0efef89c-53a3-4a48-ba70-529242dad2b8-kube-api-access-54fmr\") pod \"downloads-5bc67dcd46-skjmc\" (UID: \"0efef89c-53a3-4a48-ba70-529242dad2b8\") " pod="openshift-console/downloads-5bc67dcd46-skjmc"
Jul 17 00:17:37.142888 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.142774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxczp" event={"ID":"74ed1822-52df-4463-87a9-cd5367dfca3a","Type":"ContainerStarted","Data":"491bc2001b11ae076ea66e0aed00dd2b6b4fb1ebc634707448a2e40717508c2c"}
Jul 17 00:17:37.144225 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.144199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr" event={"ID":"8f9d1891-b7c3-42a4-951b-4145be0e8ffb","Type":"ContainerStarted","Data":"3fb6da7be74fbd036f4d4128d2a0b8b1cde4fd0887acabdc39ef447be3d0ce8d"}
Jul 17 00:17:37.144375 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.144249    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr" event={"ID":"8f9d1891-b7c3-42a4-951b-4145be0e8ffb","Type":"ContainerStarted","Data":"ce68cb778a93a48fb8d096545936787e9b0ef263982d3cc5e40779cf12099810"}
Jul 17 00:17:37.144435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.144379    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:37.145330 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.145308    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ph4x4" event={"ID":"863459a2-790d-4fab-afd5-b9a255285b3a","Type":"ContainerStarted","Data":"90188bd8ce658d08cc3330e2ba9f60c46c76671ce63142d5b1a60e7d4e1b2aa4"}
Jul 17 00:17:37.146443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.146424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xkcbl" event={"ID":"78bc1249-e4fb-49bd-8d71-ba85d657c82d","Type":"ContainerStarted","Data":"d83d698392f73aad70666b5049ccb438343373a516832c2cf730d306f40bb20e"}
Jul 17 00:17:37.166894 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.166863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54fmr\" (UniqueName: \"kubernetes.io/projected/0efef89c-53a3-4a48-ba70-529242dad2b8-kube-api-access-54fmr\") pod \"downloads-5bc67dcd46-skjmc\" (UID: \"0efef89c-53a3-4a48-ba70-529242dad2b8\") " pod="openshift-console/downloads-5bc67dcd46-skjmc"
Jul 17 00:17:37.168456 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.168415    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr" podStartSLOduration=2.168401157 podStartE2EDuration="2.168401157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:37.168023209 +0000 UTC m=+51.744175782" watchObservedRunningTime="2026-07-17 00:17:37.168401157 +0000 UTC m=+51.744553730"
Jul 17 00:17:37.192211 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.192181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54fmr\" (UniqueName: \"kubernetes.io/projected/0efef89c-53a3-4a48-ba70-529242dad2b8-kube-api-access-54fmr\") pod \"downloads-5bc67dcd46-skjmc\" (UID: \"0efef89c-53a3-4a48-ba70-529242dad2b8\") " pod="openshift-console/downloads-5bc67dcd46-skjmc"
Jul 17 00:17:37.297982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.297939    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-skjmc"
Jul 17 00:17:37.459206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.459102    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-skjmc"]
Jul 17 00:17:37.469375 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:37.469340    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efef89c_53a3_4a48_ba70_529242dad2b8.slice/crio-7b4c868112a0fa83ca35d44b24ace1a4dbbd182402378c790351249005119bd2 WatchSource:0}: Error finding container 7b4c868112a0fa83ca35d44b24ace1a4dbbd182402378c790351249005119bd2: Status 404 returned error can't find the container with id 7b4c868112a0fa83ca35d44b24ace1a4dbbd182402378c790351249005119bd2
Jul 17 00:17:37.968439 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.967750    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:37.968439 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.968214    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:37.971145 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.970895    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 00:17:37.971284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.971214    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvh75\""
Jul 17 00:17:37.971619 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.971596    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24tr\""
Jul 17 00:17:37.971864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.971846    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 00:17:37.972198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:37.972181    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 00:17:38.153861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:38.153580    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-skjmc" event={"ID":"0efef89c-53a3-4a48-ba70-529242dad2b8","Type":"ContainerStarted","Data":"7b4c868112a0fa83ca35d44b24ace1a4dbbd182402378c790351249005119bd2"}
Jul 17 00:17:42.258041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.258010    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56cffc58f5-f4w6d"]
Jul 17 00:17:42.274013 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.273266    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56cffc58f5-f4w6d"]
Jul 17 00:17:42.274013 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.273385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.276620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.275574    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 00:17:42.276620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.275918    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9gjx9\""
Jul 17 00:17:42.276620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.275977    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 00:17:42.276620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.276195    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 00:17:42.276620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.276208    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 00:17:42.276620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.276487    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 00:17:42.310923 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.310885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-oauth-config\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.311061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.310931    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-console-config\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.311061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.310956    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfl4j\" (UniqueName: \"kubernetes.io/projected/27de678a-cb1e-4495-aff6-866b479b235a-kube-api-access-tfl4j\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.311194 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.311071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-oauth-serving-cert\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.311194 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.311111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-service-ca\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.311304 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.311234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-serving-cert\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.412752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.412284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-oauth-serving-cert\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.412752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.412341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-service-ca\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.412752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.412383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-serving-cert\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.412752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.412414    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-oauth-config\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.412752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.412441    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-console-config\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.412752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.412465    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfl4j\" (UniqueName: \"kubernetes.io/projected/27de678a-cb1e-4495-aff6-866b479b235a-kube-api-access-tfl4j\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.413183 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.413049    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-oauth-serving-cert\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.413313 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.413286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-console-config\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.413376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.413325    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-service-ca\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.415510 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.415484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-serving-cert\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.415754 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.415734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-oauth-config\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.420732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.420716    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfl4j\" (UniqueName: \"kubernetes.io/projected/27de678a-cb1e-4495-aff6-866b479b235a-kube-api-access-tfl4j\") pod \"console-56cffc58f5-f4w6d\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") " pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:42.585559 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:42.585415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:17:43.891919 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:43.891863    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56cffc58f5-f4w6d"]
Jul 17 00:17:43.898397 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:43.898352    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27de678a_cb1e_4495_aff6_866b479b235a.slice/crio-0d9d3031adf2591a08c01df299d6ec085e28a595bece6ebcf308ebc9b1c17356 WatchSource:0}: Error finding container 0d9d3031adf2591a08c01df299d6ec085e28a595bece6ebcf308ebc9b1c17356: Status 404 returned error can't find the container with id 0d9d3031adf2591a08c01df299d6ec085e28a595bece6ebcf308ebc9b1c17356
Jul 17 00:17:44.174476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.174432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ph4x4" event={"ID":"863459a2-790d-4fab-afd5-b9a255285b3a","Type":"ContainerStarted","Data":"9d1ba44f5c4cec18ce38a075e91ab32943d2e65a5062aa1b758eb7df3f35538d"}
Jul 17 00:17:44.176238 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.176200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xkcbl" event={"ID":"78bc1249-e4fb-49bd-8d71-ba85d657c82d","Type":"ContainerStarted","Data":"cc875c269ede8a32601745fdd4fdff05e179095f56bb9f60020d3436a570439b"}
Jul 17 00:17:44.178353 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.178330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxczp" event={"ID":"74ed1822-52df-4463-87a9-cd5367dfca3a","Type":"ContainerStarted","Data":"82c971c75bff47cf2dcc728b21df0b43df6213237af5651c7e1a81f7423c83d9"}
Jul 17 00:17:44.178481 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.178378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxczp" event={"ID":"74ed1822-52df-4463-87a9-cd5367dfca3a","Type":"ContainerStarted","Data":"7963439bce3164c40d3d3960b35337189eb6cc5f69bb9e6fc6401265291e9038"}
Jul 17 00:17:44.178481 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.178431    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:44.179856 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.179832    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cffc58f5-f4w6d" event={"ID":"27de678a-cb1e-4495-aff6-866b479b235a","Type":"ContainerStarted","Data":"0d9d3031adf2591a08c01df299d6ec085e28a595bece6ebcf308ebc9b1c17356"}
Jul 17 00:17:44.189169 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.189119    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ph4x4" podStartSLOduration=1.818512346 podStartE2EDuration="8.189105793s" totalImagesPullingTime="6.370593447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:44.188366256 +0000 UTC m=+58.764518831" watchObservedRunningTime="2026-07-17 00:17:44.189105793 +0000 UTC m=+58.765258368"
Jul 17 00:17:44.204504 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:44.204300    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zxczp" podStartSLOduration=1.779481415 podStartE2EDuration="8.204279328s" totalImagesPullingTime="6.424797913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:44.203310347 +0000 UTC m=+58.779462923" watchObservedRunningTime="2026-07-17 00:17:44.204279328 +0000 UTC m=+58.780431904"
Jul 17 00:17:48.104205 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:48.103850    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-645g2"
Jul 17 00:17:51.594252 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.594198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:51.596720 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.596663    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 00:17:51.609097 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.609027    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb595638-7449-478a-8505-2e0e531829a1-original-pull-secret\") pod \"global-pull-secret-syncer-pfsxc\" (UID: \"fb595638-7449-478a-8505-2e0e531829a1\") " pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:51.691859 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.691816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfsxc"
Jul 17 00:17:51.695049 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.694709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:51.697107 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.697082    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 00:17:51.708388 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.708356    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1bc817-d1b9-458a-a933-8af8ce5e95b3-metrics-certs\") pod \"network-metrics-daemon-vjwrp\" (UID: \"7e1bc817-d1b9-458a-a933-8af8ce5e95b3\") " pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:51.790226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.790182    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvh75\""
Jul 17 00:17:51.796044 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.796005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:51.797748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.797728    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 00:17:51.798772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.798752    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjwrp"
Jul 17 00:17:51.809048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.809013    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 00:17:51.820902 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:51.820874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6z2f\" (UniqueName: \"kubernetes.io/projected/51ed8803-1a24-4ff5-8f3f-31b1ccad98ed-kube-api-access-n6z2f\") pod \"network-check-target-7nrcr\" (UID: \"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed\") " pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:52.100300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:52.100259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24tr\""
Jul 17 00:17:52.108822 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:52.108779    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:17:54.187749 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:54.187711    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zxczp"
Jul 17 00:17:56.281563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.281524    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-snsws"]
Jul 17 00:17:56.294368 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.294335    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.294542 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.294521    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-snsws"]
Jul 17 00:17:56.296290 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.296263    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 17 00:17:56.296747 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.296689    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 00:17:56.298301 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.297210    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qgp6v\""
Jul 17 00:17:56.298301 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.296696    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 17 00:17:56.298301 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.297772    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 00:17:56.298301 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.297981    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 00:17:56.434732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.434678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.434732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.434737    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.434972 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.434852    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qf7\" (UniqueName: \"kubernetes.io/projected/a26c5aaa-9bc1-4a94-8687-386cb8004934-kube-api-access-f2qf7\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.434972 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.434913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a26c5aaa-9bc1-4a94-8687-386cb8004934-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.535546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.535457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.535546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.535529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.535900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.535582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qf7\" (UniqueName: \"kubernetes.io/projected/a26c5aaa-9bc1-4a94-8687-386cb8004934-kube-api-access-f2qf7\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.535900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.535713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a26c5aaa-9bc1-4a94-8687-386cb8004934-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.536014 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:56.535586    2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 17 00:17:56.536014 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:17:56.535980    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-tls podName:a26c5aaa-9bc1-4a94-8687-386cb8004934 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:57.035955639 +0000 UTC m=+71.612108197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-snsws" (UID: "a26c5aaa-9bc1-4a94-8687-386cb8004934") : secret "prometheus-operator-tls" not found
Jul 17 00:17:56.536433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.536411    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a26c5aaa-9bc1-4a94-8687-386cb8004934-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.538261 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.538237    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.543717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.543697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qf7\" (UniqueName: \"kubernetes.io/projected/a26c5aaa-9bc1-4a94-8687-386cb8004934-kube-api-access-f2qf7\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:56.653814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.651303    2573 patch_prober.go:28] interesting pod/image-registry-6c97cc46cb-jmhcr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 00:17:56.653814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.651372    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr" podUID="8f9d1891-b7c3-42a4-951b-4145be0e8ffb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 00:17:56.759674 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.759549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pfsxc"]
Jul 17 00:17:56.765018 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:56.764972    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb595638_7449_478a_8505_2e0e531829a1.slice/crio-dc6e0be96ddb93d8b548a3df3deff11be8ae4313f3f8d89dea53eb4f5258a682 WatchSource:0}: Error finding container dc6e0be96ddb93d8b548a3df3deff11be8ae4313f3f8d89dea53eb4f5258a682: Status 404 returned error can't find the container with id dc6e0be96ddb93d8b548a3df3deff11be8ae4313f3f8d89dea53eb4f5258a682
Jul 17 00:17:56.778743 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.778712    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7nrcr"]
Jul 17 00:17:56.779754 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:56.779728    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vjwrp"]
Jul 17 00:17:56.783496 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:56.783455    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ed8803_1a24_4ff5_8f3f_31b1ccad98ed.slice/crio-4992a827b015d5882548fadb0639f6534b869b57f151752ce60ea6942f1f251f WatchSource:0}: Error finding container 4992a827b015d5882548fadb0639f6534b869b57f151752ce60ea6942f1f251f: Status 404 returned error can't find the container with id 4992a827b015d5882548fadb0639f6534b869b57f151752ce60ea6942f1f251f
Jul 17 00:17:56.783892 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:17:56.783862    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1bc817_d1b9_458a_a933_8af8ce5e95b3.slice/crio-ab8e183dc409319f283702c6380c3aa6237158bea48e8e244ce2a2122e36dee0 WatchSource:0}: Error finding container ab8e183dc409319f283702c6380c3aa6237158bea48e8e244ce2a2122e36dee0: Status 404 returned error can't find the container with id ab8e183dc409319f283702c6380c3aa6237158bea48e8e244ce2a2122e36dee0
Jul 17 00:17:57.040088 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.039986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:57.042960 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.042920    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26c5aaa-9bc1-4a94-8687-386cb8004934-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-snsws\" (UID: \"a26c5aaa-9bc1-4a94-8687-386cb8004934\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:57.209090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.208741    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws"
Jul 17 00:17:57.245196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.244315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cffc58f5-f4w6d" event={"ID":"27de678a-cb1e-4495-aff6-866b479b235a","Type":"ContainerStarted","Data":"cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3"}
Jul 17 00:17:57.251574 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.251486    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7nrcr" event={"ID":"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed","Type":"ContainerStarted","Data":"4992a827b015d5882548fadb0639f6534b869b57f151752ce60ea6942f1f251f"}
Jul 17 00:17:57.255273 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.255177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xkcbl" event={"ID":"78bc1249-e4fb-49bd-8d71-ba85d657c82d","Type":"ContainerStarted","Data":"b1fd8c549971cec1285b3f176cf7764b7234a2492fef9d0712954407607a9b1a"}
Jul 17 00:17:57.259112 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.259020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pfsxc" event={"ID":"fb595638-7449-478a-8505-2e0e531829a1","Type":"ContainerStarted","Data":"dc6e0be96ddb93d8b548a3df3deff11be8ae4313f3f8d89dea53eb4f5258a682"}
Jul 17 00:17:57.264214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.263907    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56cffc58f5-f4w6d" podStartSLOduration=2.588973531 podStartE2EDuration="15.263885268s" totalImagesPullingTime="12.674911737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:57.263080603 +0000 UTC m=+71.839233178" watchObservedRunningTime="2026-07-17 00:17:57.263885268 +0000 UTC m=+71.840037843"
Jul 17 00:17:57.266143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.266038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vjwrp" event={"ID":"7e1bc817-d1b9-458a-a933-8af8ce5e95b3","Type":"ContainerStarted","Data":"ab8e183dc409319f283702c6380c3aa6237158bea48e8e244ce2a2122e36dee0"}
Jul 17 00:17:57.271436 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.269234    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-skjmc" event={"ID":"0efef89c-53a3-4a48-ba70-529242dad2b8","Type":"ContainerStarted","Data":"2dfee93ae8d3cb6701772e3a98f3ded130fb95e03b64e8b8641659ec9ab54637"}
Jul 17 00:17:57.272862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.272835    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-skjmc"
Jul 17 00:17:57.279671 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.279646    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-skjmc"
Jul 17 00:17:57.311899 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.311473    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-skjmc" podStartSLOduration=2.142721292 podStartE2EDuration="21.311449613s" totalImagesPullingTime="19.168728321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:57.293124383 +0000 UTC m=+71.869276959" watchObservedRunningTime="2026-07-17 00:17:57.311449613 +0000 UTC m=+71.887602189"
Jul 17 00:17:57.411688 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:57.410830    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-snsws"]
Jul 17 00:17:58.160954 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:58.160912    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c97cc46cb-jmhcr"
Jul 17 00:17:58.280148 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:17:58.280099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws" event={"ID":"a26c5aaa-9bc1-4a94-8687-386cb8004934","Type":"ContainerStarted","Data":"444dc4f02cad91d32663a513e04ee4cd07c4fed66f5c7f271ff4b730764c689c"}
Jul 17 00:18:01.315745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:01.315537    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vjwrp" event={"ID":"7e1bc817-d1b9-458a-a933-8af8ce5e95b3","Type":"ContainerStarted","Data":"488fcf65fbec8a94c07da51a04d53a65985ccac997637cc887b271743187f9ca"}
Jul 17 00:18:01.315745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:01.315680    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vjwrp" event={"ID":"7e1bc817-d1b9-458a-a933-8af8ce5e95b3","Type":"ContainerStarted","Data":"1f899dc6d94363f792e52434fa75c0c278543a435679b99aef4d9f5cf5dadd76"}
Jul 17 00:18:01.320292 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:01.319686    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws" event={"ID":"a26c5aaa-9bc1-4a94-8687-386cb8004934","Type":"ContainerStarted","Data":"abd9e906a1351daaadc12802d36be161d0b87f8fe7e6aea46f88d7aa10fd9e8e"}
Jul 17 00:18:01.320292 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:01.319744    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws" event={"ID":"a26c5aaa-9bc1-4a94-8687-386cb8004934","Type":"ContainerStarted","Data":"327eeb8dae34fd8955798ee2fbd09bdc6d6a2a7121573b92e2ab24b1429df8ea"}
Jul 17 00:18:01.356717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:01.356648    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-snsws" podStartSLOduration=1.8716242140000001 podStartE2EDuration="5.356623544s" totalImagesPullingTime="3.48499933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:01.355256025 +0000 UTC m=+75.931408601" watchObservedRunningTime="2026-07-17 00:18:01.356623544 +0000 UTC m=+75.932776121"
Jul 17 00:18:01.358065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:01.358004    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vjwrp" podStartSLOduration=72.650490919 podStartE2EDuration="1m16.357987843s" totalImagesPullingTime="3.707496924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:01.336599235 +0000 UTC m=+75.912751810" watchObservedRunningTime="2026-07-17 00:18:01.357987843 +0000 UTC m=+75.934140420"
Jul 17 00:18:02.488158 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:02.488115    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56cffc58f5-f4w6d"]
Jul 17 00:18:02.585625 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:02.585585    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:18:03.626870 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.626824    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f99ft"]
Jul 17 00:18:03.716551 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.715611    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.718691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.717940    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 00:18:03.718691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.718058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2g2fw\""
Jul 17 00:18:03.718691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.718215    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 00:18:03.718691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.718326    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 00:18:03.817473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-textfile\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.817473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-tls\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.817473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vf9\" (UniqueName: \"kubernetes.io/projected/54fc07a8-3185-4927-b5d9-8e39e973f51d-kube-api-access-26vf9\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.817473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817495    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-sys\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.817473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817527    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.818005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817560    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.818005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817629    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fc07a8-3185-4927-b5d9-8e39e973f51d-metrics-client-ca\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.818005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817661    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-wtmp\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.818005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.817691    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-root\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.918772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.918666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vf9\" (UniqueName: \"kubernetes.io/projected/54fc07a8-3185-4927-b5d9-8e39e973f51d-kube-api-access-26vf9\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.918844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-sys\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.918886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.918926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.918957    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fc07a8-3185-4927-b5d9-8e39e973f51d-metrics-client-ca\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.918976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-wtmp\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.918999    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-root\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.919025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-textfile\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.919080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-tls\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.919288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.919141    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-root\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.920718 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.919304    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-wtmp\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.920718 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.919331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fc07a8-3185-4927-b5d9-8e39e973f51d-sys\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.920718 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.919627    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.920718 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.919809    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-textfile\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.920718 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.920338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fc07a8-3185-4927-b5d9-8e39e973f51d-metrics-client-ca\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.922687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.922662    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.923681 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.923649    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fc07a8-3185-4927-b5d9-8e39e973f51d-node-exporter-tls\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:03.926604 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:03.926558    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vf9\" (UniqueName: \"kubernetes.io/projected/54fc07a8-3185-4927-b5d9-8e39e973f51d-kube-api-access-26vf9\") pod \"node-exporter-f99ft\" (UID: \"54fc07a8-3185-4927-b5d9-8e39e973f51d\") " pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:04.029614 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:04.029564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f99ft"
Jul 17 00:18:06.993439 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:18:06.993409    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fc07a8_3185_4927_b5d9_8e39e973f51d.slice/crio-ddc425cad5ea29349741f8f98cd841309891de0ab2f90fdd7077ee793bb9d6a7 WatchSource:0}: Error finding container ddc425cad5ea29349741f8f98cd841309891de0ab2f90fdd7077ee793bb9d6a7: Status 404 returned error can't find the container with id ddc425cad5ea29349741f8f98cd841309891de0ab2f90fdd7077ee793bb9d6a7
Jul 17 00:18:07.349986 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.349857    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7nrcr" event={"ID":"51ed8803-1a24-4ff5-8f3f-31b1ccad98ed","Type":"ContainerStarted","Data":"1ed0ac0867d75de487c184fdcab77b120b48a9e6345eb1cea1e398d5accc2f0e"}
Jul 17 00:18:07.350202 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.350002    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:18:07.352216 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.352178    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xkcbl" event={"ID":"78bc1249-e4fb-49bd-8d71-ba85d657c82d","Type":"ContainerStarted","Data":"c75ecfc56b30141be5c492f1cc4170bcdfd336ae031167bbaebd7ec9018da362"}
Jul 17 00:18:07.353632 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.353608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pfsxc" event={"ID":"fb595638-7449-478a-8505-2e0e531829a1","Type":"ContainerStarted","Data":"1bce0fcda76dbc611a4be52707fdf082804471dae5ad311b6238c0b49e0f72f6"}
Jul 17 00:18:07.354875 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.354852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f99ft" event={"ID":"54fc07a8-3185-4927-b5d9-8e39e973f51d","Type":"ContainerStarted","Data":"ddc425cad5ea29349741f8f98cd841309891de0ab2f90fdd7077ee793bb9d6a7"}
Jul 17 00:18:07.364989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.364948    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7nrcr" podStartSLOduration=72.161001899 podStartE2EDuration="1m22.36493408s" totalImagesPullingTime="10.203932181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:07.364010727 +0000 UTC m=+81.940163304" watchObservedRunningTime="2026-07-17 00:18:07.36493408 +0000 UTC m=+81.941086654"
Jul 17 00:18:07.377285 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.377232    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pfsxc" podStartSLOduration=72.140302088 podStartE2EDuration="1m22.377216234s" totalImagesPullingTime="10.236914146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:07.37627991 +0000 UTC m=+81.952432485" watchObservedRunningTime="2026-07-17 00:18:07.377216234 +0000 UTC m=+81.953368825"
Jul 17 00:18:07.391923 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.391835    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xkcbl" podStartSLOduration=1.555409276 podStartE2EDuration="31.391811423s" totalImagesPullingTime="29.836402147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:07.39101763 +0000 UTC m=+81.967170205" watchObservedRunningTime="2026-07-17 00:18:07.391811423 +0000 UTC m=+81.967963988"
Jul 17 00:18:07.923046 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.922890    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5fdc7b5777-jfptx"]
Jul 17 00:18:07.950983 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.950946    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fdc7b5777-jfptx"]
Jul 17 00:18:07.951159 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.951069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:07.953389 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.953358    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 00:18:07.953551 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.953399    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 17 00:18:07.953551 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.953487    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s6qpf\""
Jul 17 00:18:07.953861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.953842    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 17 00:18:07.953991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.953972    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 17 00:18:07.954071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:07.954058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1r626p03kr7f0\""
Jul 17 00:18:08.056544 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.056504    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-secret-metrics-server-tls\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.056544 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.056549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-client-ca-bundle\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.057019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.056603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wtf\" (UniqueName: \"kubernetes.io/projected/06aaf4d6-823e-416f-a28a-29877e7c8223-kube-api-access-26wtf\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.057019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.056707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/06aaf4d6-823e-416f-a28a-29877e7c8223-audit-log\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.057019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.056743    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-secret-metrics-server-client-certs\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.057019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.056777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06aaf4d6-823e-416f-a28a-29877e7c8223-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.057019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.056828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/06aaf4d6-823e-416f-a28a-29877e7c8223-metrics-server-audit-profiles\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.157553 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.157516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/06aaf4d6-823e-416f-a28a-29877e7c8223-metrics-server-audit-profiles\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.157775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.157572    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-secret-metrics-server-tls\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.157775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.157614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-client-ca-bundle\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.157775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.157645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26wtf\" (UniqueName: \"kubernetes.io/projected/06aaf4d6-823e-416f-a28a-29877e7c8223-kube-api-access-26wtf\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.157775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.157677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/06aaf4d6-823e-416f-a28a-29877e7c8223-audit-log\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.157775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.157708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-secret-metrics-server-client-certs\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.157775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.157748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06aaf4d6-823e-416f-a28a-29877e7c8223-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.158106 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.158087    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/06aaf4d6-823e-416f-a28a-29877e7c8223-audit-log\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.158864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.158834    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06aaf4d6-823e-416f-a28a-29877e7c8223-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.159050 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.158874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/06aaf4d6-823e-416f-a28a-29877e7c8223-metrics-server-audit-profiles\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.160488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.160459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-secret-metrics-server-tls\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.160618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.160525    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-client-ca-bundle\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.160618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.160535    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/06aaf4d6-823e-416f-a28a-29877e7c8223-secret-metrics-server-client-certs\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.168561 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.168530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wtf\" (UniqueName: \"kubernetes.io/projected/06aaf4d6-823e-416f-a28a-29877e7c8223-kube-api-access-26wtf\") pod \"metrics-server-5fdc7b5777-jfptx\" (UID: \"06aaf4d6-823e-416f-a28a-29877e7c8223\") " pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.260688 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.260645    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:08.459389 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.459355    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"]
Jul 17 00:18:08.485188 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.485146    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"]
Jul 17 00:18:08.485312 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.485270    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:08.488072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.488041    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 17 00:18:08.488204 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.488137    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wdnkn\""
Jul 17 00:18:08.501149 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.501115    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fdc7b5777-jfptx"]
Jul 17 00:18:08.505750 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:18:08.505718    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06aaf4d6_823e_416f_a28a_29877e7c8223.slice/crio-6bd4ff2825b05cbe8c9c1c4d072ca7dae7a8ad47399f3999d5e752d1bb26cbb6 WatchSource:0}: Error finding container 6bd4ff2825b05cbe8c9c1c4d072ca7dae7a8ad47399f3999d5e752d1bb26cbb6: Status 404 returned error can't find the container with id 6bd4ff2825b05cbe8c9c1c4d072ca7dae7a8ad47399f3999d5e752d1bb26cbb6
Jul 17 00:18:08.661171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.661132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqzhw\" (UniqueName: \"kubernetes.io/projected/9b9f177b-9904-4f5a-9610-8c9d79745ad8-kube-api-access-dqzhw\") pod \"monitoring-plugin-7b7b7bc8bf-8tnv5\" (UID: \"9b9f177b-9904-4f5a-9610-8c9d79745ad8\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:08.661332 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.661189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b9f177b-9904-4f5a-9610-8c9d79745ad8-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8tnv5\" (UID: \"9b9f177b-9904-4f5a-9610-8c9d79745ad8\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:08.761855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.761783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqzhw\" (UniqueName: \"kubernetes.io/projected/9b9f177b-9904-4f5a-9610-8c9d79745ad8-kube-api-access-dqzhw\") pod \"monitoring-plugin-7b7b7bc8bf-8tnv5\" (UID: \"9b9f177b-9904-4f5a-9610-8c9d79745ad8\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:08.762050 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.761903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b9f177b-9904-4f5a-9610-8c9d79745ad8-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8tnv5\" (UID: \"9b9f177b-9904-4f5a-9610-8c9d79745ad8\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:08.762050 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:18:08.762003    2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 17 00:18:08.762183 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:18:08.762070    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b9f177b-9904-4f5a-9610-8c9d79745ad8-monitoring-plugin-cert podName:9b9f177b-9904-4f5a-9610-8c9d79745ad8 nodeName:}" failed. No retries permitted until 2026-07-17 00:18:09.262047784 +0000 UTC m=+83.838200337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9b9f177b-9904-4f5a-9610-8c9d79745ad8-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-8tnv5" (UID: "9b9f177b-9904-4f5a-9610-8c9d79745ad8") : secret "monitoring-plugin-cert" not found
Jul 17 00:18:08.769645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:08.769616    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqzhw\" (UniqueName: \"kubernetes.io/projected/9b9f177b-9904-4f5a-9610-8c9d79745ad8-kube-api-access-dqzhw\") pod \"monitoring-plugin-7b7b7bc8bf-8tnv5\" (UID: \"9b9f177b-9904-4f5a-9610-8c9d79745ad8\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:09.268416 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:09.268370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b9f177b-9904-4f5a-9610-8c9d79745ad8-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8tnv5\" (UID: \"9b9f177b-9904-4f5a-9610-8c9d79745ad8\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:09.271017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:09.270986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b9f177b-9904-4f5a-9610-8c9d79745ad8-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8tnv5\" (UID: \"9b9f177b-9904-4f5a-9610-8c9d79745ad8\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:09.364723 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:09.364687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx" event={"ID":"06aaf4d6-823e-416f-a28a-29877e7c8223","Type":"ContainerStarted","Data":"6bd4ff2825b05cbe8c9c1c4d072ca7dae7a8ad47399f3999d5e752d1bb26cbb6"}
Jul 17 00:18:09.366952 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:09.366916    2573 generic.go:356] "Generic (PLEG): container finished" podID="54fc07a8-3185-4927-b5d9-8e39e973f51d" containerID="727217ec16303162bbf2d2b23d421ed01063e9b3f417f96147287baf119b97a0" exitCode=0
Jul 17 00:18:09.367130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:09.366973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f99ft" event={"ID":"54fc07a8-3185-4927-b5d9-8e39e973f51d","Type":"ContainerDied","Data":"727217ec16303162bbf2d2b23d421ed01063e9b3f417f96147287baf119b97a0"}
Jul 17 00:18:09.395189 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:09.395155    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:09.570970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:09.570915    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"]
Jul 17 00:18:09.574613 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:18:09.574586    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9f177b_9904_4f5a_9610_8c9d79745ad8.slice/crio-85c6253fd31b1bb132fa9ab1930ddf831694b35764d58a8fbaa0413c0769fb51 WatchSource:0}: Error finding container 85c6253fd31b1bb132fa9ab1930ddf831694b35764d58a8fbaa0413c0769fb51: Status 404 returned error can't find the container with id 85c6253fd31b1bb132fa9ab1930ddf831694b35764d58a8fbaa0413c0769fb51
Jul 17 00:18:10.372850 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:10.372810    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5" event={"ID":"9b9f177b-9904-4f5a-9610-8c9d79745ad8","Type":"ContainerStarted","Data":"85c6253fd31b1bb132fa9ab1930ddf831694b35764d58a8fbaa0413c0769fb51"}
Jul 17 00:18:10.374854 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:10.374817    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f99ft" event={"ID":"54fc07a8-3185-4927-b5d9-8e39e973f51d","Type":"ContainerStarted","Data":"ad34800ec63f1db85754466a81a263ebfd5cac0e8cc5f2f31b4daa6df76395ab"}
Jul 17 00:18:10.374979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:10.374880    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f99ft" event={"ID":"54fc07a8-3185-4927-b5d9-8e39e973f51d","Type":"ContainerStarted","Data":"56ebdce81f4188ea3ad92c430ccf54b8bd7a7ec4e657f72c61a141a15d01a25f"}
Jul 17 00:18:10.393905 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:10.393822    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f99ft" podStartSLOduration=5.953552392 podStartE2EDuration="7.393782391s" totalImagesPullingTime="1.34016659s" totalInitContainerRuntime="100.063409ms" isStatefulPod=true podCreationTimestamp="2026-07-17 00:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:10.392771053 +0000 UTC m=+84.968923627" watchObservedRunningTime="2026-07-17 00:18:10.393782391 +0000 UTC m=+84.969934967"
Jul 17 00:18:11.381981 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:11.381933    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx" event={"ID":"06aaf4d6-823e-416f-a28a-29877e7c8223","Type":"ContainerStarted","Data":"53a51f6653fd053ae9619e312befa6327ce6186cac0957db4de547da85be346a"}
Jul 17 00:18:11.398314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:11.398246    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx" podStartSLOduration=2.402332894 podStartE2EDuration="4.398225236s" totalImagesPullingTime="1.995892342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:11.397255709 +0000 UTC m=+85.973408283" watchObservedRunningTime="2026-07-17 00:18:11.398225236 +0000 UTC m=+85.974377868"
Jul 17 00:18:12.387155 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:12.387111    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5" event={"ID":"9b9f177b-9904-4f5a-9610-8c9d79745ad8","Type":"ContainerStarted","Data":"d4ed106a5f23820f186a0f5b467994cfb24dba4eb47de87f31e8975d9f2c48d5"}
Jul 17 00:18:12.387676 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:12.387532    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:12.392181 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:12.392150    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5"
Jul 17 00:18:12.402523 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:12.402473    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8tnv5" podStartSLOduration=2.109885322 podStartE2EDuration="4.402457549s" totalImagesPullingTime="2.292572227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:12.401154254 +0000 UTC m=+86.977306827" watchObservedRunningTime="2026-07-17 00:18:12.402457549 +0000 UTC m=+86.978610122"
Jul 17 00:18:23.587581 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:23.587537    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ph4x4_863459a2-790d-4fab-afd5-b9a255285b3a/serve-healthcheck-canary/0.log"
Jul 17 00:18:27.518289 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.518239    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56cffc58f5-f4w6d" podUID="27de678a-cb1e-4495-aff6-866b479b235a" containerName="console" containerID="cri-o://cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3" gracePeriod=15
Jul 17 00:18:27.776006 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.775935    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:18:27.809147 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809114    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-serving-cert\") pod \"27de678a-cb1e-4495-aff6-866b479b235a\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") "
Jul 17 00:18:27.809318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809177    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-service-ca\") pod \"27de678a-cb1e-4495-aff6-866b479b235a\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") "
Jul 17 00:18:27.809318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809223    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-oauth-config\") pod \"27de678a-cb1e-4495-aff6-866b479b235a\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") "
Jul 17 00:18:27.809318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809248    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-oauth-serving-cert\") pod \"27de678a-cb1e-4495-aff6-866b479b235a\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") "
Jul 17 00:18:27.809318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809278    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-console-config\") pod \"27de678a-cb1e-4495-aff6-866b479b235a\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") "
Jul 17 00:18:27.809531 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809331    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfl4j\" (UniqueName: \"kubernetes.io/projected/27de678a-cb1e-4495-aff6-866b479b235a-kube-api-access-tfl4j\") pod \"27de678a-cb1e-4495-aff6-866b479b235a\" (UID: \"27de678a-cb1e-4495-aff6-866b479b235a\") "
Jul 17 00:18:27.809903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809696    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-service-ca" (OuterVolumeSpecName: "service-ca") pod "27de678a-cb1e-4495-aff6-866b479b235a" (UID: "27de678a-cb1e-4495-aff6-866b479b235a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:18:27.809903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809834    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "27de678a-cb1e-4495-aff6-866b479b235a" (UID: "27de678a-cb1e-4495-aff6-866b479b235a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:18:27.809903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.809876    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-console-config" (OuterVolumeSpecName: "console-config") pod "27de678a-cb1e-4495-aff6-866b479b235a" (UID: "27de678a-cb1e-4495-aff6-866b479b235a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:18:27.811866 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.811834    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "27de678a-cb1e-4495-aff6-866b479b235a" (UID: "27de678a-cb1e-4495-aff6-866b479b235a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:18:27.811973 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.811888    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27de678a-cb1e-4495-aff6-866b479b235a-kube-api-access-tfl4j" (OuterVolumeSpecName: "kube-api-access-tfl4j") pod "27de678a-cb1e-4495-aff6-866b479b235a" (UID: "27de678a-cb1e-4495-aff6-866b479b235a"). InnerVolumeSpecName "kube-api-access-tfl4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:18:27.812021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.812002    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "27de678a-cb1e-4495-aff6-866b479b235a" (UID: "27de678a-cb1e-4495-aff6-866b479b235a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:18:27.909957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.909922    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfl4j\" (UniqueName: \"kubernetes.io/projected/27de678a-cb1e-4495-aff6-866b479b235a-kube-api-access-tfl4j\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:18:27.909957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.909950    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:18:27.909957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.909959    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:18:27.909957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.909967    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27de678a-cb1e-4495-aff6-866b479b235a-console-oauth-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:18:27.910218 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.909977    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-oauth-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:18:27.910218 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:27.909985    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27de678a-cb1e-4495-aff6-866b479b235a-console-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:18:28.261547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.261521    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:28.261695 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.261575    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:28.444575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.444543    2573 generic.go:356] "Generic (PLEG): container finished" podID="27de678a-cb1e-4495-aff6-866b479b235a" containerID="cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3" exitCode=0
Jul 17 00:18:28.444712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.444595    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cffc58f5-f4w6d"
Jul 17 00:18:28.444712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.444601    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cffc58f5-f4w6d" event={"ID":"27de678a-cb1e-4495-aff6-866b479b235a","Type":"ContainerDied","Data":"cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3"}
Jul 17 00:18:28.444712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.444644    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cffc58f5-f4w6d" event={"ID":"27de678a-cb1e-4495-aff6-866b479b235a","Type":"ContainerDied","Data":"0d9d3031adf2591a08c01df299d6ec085e28a595bece6ebcf308ebc9b1c17356"}
Jul 17 00:18:28.444712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.444665    2573 scope.go:117] "RemoveContainer" containerID="cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3"
Jul 17 00:18:28.452864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.452837    2573 scope.go:117] "RemoveContainer" containerID="cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3"
Jul 17 00:18:28.453115 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:18:28.453094    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3\": container with ID starting with cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3 not found: ID does not exist" containerID="cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3"
Jul 17 00:18:28.453168 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.453122    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3"} err="failed to get container status \"cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3\": rpc error: code = NotFound desc = could not find container \"cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3\": container with ID starting with cc017f7b4662a07549a783a809dea9eb76ab4dd37643bbd670263952ca51f4b3 not found: ID does not exist"
Jul 17 00:18:28.458346 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.458324    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56cffc58f5-f4w6d"]
Jul 17 00:18:28.461527 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:28.461509    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56cffc58f5-f4w6d"]
Jul 17 00:18:29.966708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:29.966660    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27de678a-cb1e-4495-aff6-866b479b235a" path="/var/lib/kubelet/pods/27de678a-cb1e-4495-aff6-866b479b235a/volumes"
Jul 17 00:18:38.373657 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:38.373620    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7nrcr"
Jul 17 00:18:48.266778 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:48.266748    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:18:48.270633 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:18:48.270604    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5fdc7b5777-jfptx"
Jul 17 00:19:31.425532 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.425490    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-647467766c-jq7gg"]
Jul 17 00:19:31.426032 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.425749    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27de678a-cb1e-4495-aff6-866b479b235a" containerName="console"
Jul 17 00:19:31.426032 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.425760    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="27de678a-cb1e-4495-aff6-866b479b235a" containerName="console"
Jul 17 00:19:31.426032 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.425849    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="27de678a-cb1e-4495-aff6-866b479b235a" containerName="console"
Jul 17 00:19:31.428154 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.428136    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.430376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.430347    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 00:19:31.430526 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.430347    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 00:19:31.431011 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.430990    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 00:19:31.431120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.431026    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 00:19:31.431275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.431259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 00:19:31.431488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.431472    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9gjx9\""
Jul 17 00:19:31.436678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.436652    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 00:19:31.439284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.439247    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647467766c-jq7gg"]
Jul 17 00:19:31.536819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.536759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-oauth-config\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.537071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.536831    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmrt\" (UniqueName: \"kubernetes.io/projected/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-kube-api-access-5zmrt\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.537071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.536894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-config\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.537071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.536943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-service-ca\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.537071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.536979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-serving-cert\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.537071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.537003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-oauth-serving-cert\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.537071 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.537020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-trusted-ca-bundle\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-oauth-serving-cert\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-trusted-ca-bundle\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638331 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-oauth-config\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638331 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmrt\" (UniqueName: \"kubernetes.io/projected/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-kube-api-access-5zmrt\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638331 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-config\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638494 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-service-ca\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638550 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-serving-cert\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.638980 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638946    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-oauth-serving-cert\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.639091 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.638960    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-config\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.639091 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.639061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-trusted-ca-bundle\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.639181 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.639163    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-service-ca\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.640711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.640693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-oauth-config\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.640908 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.640891    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-serving-cert\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.645400 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.645374    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmrt\" (UniqueName: \"kubernetes.io/projected/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-kube-api-access-5zmrt\") pod \"console-647467766c-jq7gg\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") " pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.740080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.739981    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:31.877716 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:31.877617    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647467766c-jq7gg"]
Jul 17 00:19:31.880440 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:19:31.880409    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eea02e1_aeb2_48eb_89c6_b6d11fb3b875.slice/crio-d1bb4d7cdb9182889df2d9351333409a23085c3a1474edc24546d453d3ae2007 WatchSource:0}: Error finding container d1bb4d7cdb9182889df2d9351333409a23085c3a1474edc24546d453d3ae2007: Status 404 returned error can't find the container with id d1bb4d7cdb9182889df2d9351333409a23085c3a1474edc24546d453d3ae2007
Jul 17 00:19:32.659297 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:32.659257    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647467766c-jq7gg" event={"ID":"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875","Type":"ContainerStarted","Data":"2707bce0e149c638369fd572071ed96d48cab979d4b954f2183096ba5f449459"}
Jul 17 00:19:32.659745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:32.659311    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647467766c-jq7gg" event={"ID":"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875","Type":"ContainerStarted","Data":"d1bb4d7cdb9182889df2d9351333409a23085c3a1474edc24546d453d3ae2007"}
Jul 17 00:19:32.676290 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:32.676233    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-647467766c-jq7gg" podStartSLOduration=1.6762144860000001 podStartE2EDuration="1.676214486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:19:32.675021269 +0000 UTC m=+167.251173844" watchObservedRunningTime="2026-07-17 00:19:32.676214486 +0000 UTC m=+167.252367060"
Jul 17 00:19:41.740240 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:41.740141    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:41.740752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:41.740254    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:41.745187 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:41.745161    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:19:42.702485 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:19:42.702452    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:20:17.181003 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.180961    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"]
Jul 17 00:20:17.183490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.183470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.185521 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.185494    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 00:20:17.185667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.185593    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 00:20:17.186006 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.185991    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nt95l\""
Jul 17 00:20:17.192084 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.192060    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"]
Jul 17 00:20:17.271317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.271282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.271317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.271325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.271552 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.271371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqrr\" (UniqueName: \"kubernetes.io/projected/bbee3c66-965d-4f2a-9432-a090d34ca64a-kube-api-access-qwqrr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.372767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.372708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.373016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.372832    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqrr\" (UniqueName: \"kubernetes.io/projected/bbee3c66-965d-4f2a-9432-a090d34ca64a-kube-api-access-qwqrr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.373016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.372863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.373244 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.373219    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.373314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.373227    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.380759 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.380732    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqrr\" (UniqueName: \"kubernetes.io/projected/bbee3c66-965d-4f2a-9432-a090d34ca64a-kube-api-access-qwqrr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.493217 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.493114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:17.629953 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.629918    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"]
Jul 17 00:20:17.633069 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:20:17.633040    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbee3c66_965d_4f2a_9432_a090d34ca64a.slice/crio-c2d7a52ca7551075f74f1f5af1248ae81844d429af8c62568fb096f4370dfcf6 WatchSource:0}: Error finding container c2d7a52ca7551075f74f1f5af1248ae81844d429af8c62568fb096f4370dfcf6: Status 404 returned error can't find the container with id c2d7a52ca7551075f74f1f5af1248ae81844d429af8c62568fb096f4370dfcf6
Jul 17 00:20:17.821058 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:17.821014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf" event={"ID":"bbee3c66-965d-4f2a-9432-a090d34ca64a","Type":"ContainerStarted","Data":"c2d7a52ca7551075f74f1f5af1248ae81844d429af8c62568fb096f4370dfcf6"}
Jul 17 00:20:25.854803 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:25.854755    2573 generic.go:356] "Generic (PLEG): container finished" podID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerID="1415e924ceea25bec0445858bc3cbe7e2faf89b2cbefaaa4b06a19c264bc8103" exitCode=0
Jul 17 00:20:25.855137 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:25.854826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf" event={"ID":"bbee3c66-965d-4f2a-9432-a090d34ca64a","Type":"ContainerDied","Data":"1415e924ceea25bec0445858bc3cbe7e2faf89b2cbefaaa4b06a19c264bc8103"}
Jul 17 00:20:28.869047 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:28.869005    2573 generic.go:356] "Generic (PLEG): container finished" podID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerID="0ec4bf3a8d288414ec1b4c43c766f02085621443458d8129571c36ad4e50d0ba" exitCode=0
Jul 17 00:20:28.869537 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:28.869085    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf" event={"ID":"bbee3c66-965d-4f2a-9432-a090d34ca64a","Type":"ContainerDied","Data":"0ec4bf3a8d288414ec1b4c43c766f02085621443458d8129571c36ad4e50d0ba"}
Jul 17 00:20:36.908664 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:36.908618    2573 generic.go:356] "Generic (PLEG): container finished" podID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerID="c90fde3c50a9f2ea99fb48ebcda8044087bd0c2346389b3014cc45fcac67c639" exitCode=0
Jul 17 00:20:36.909061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:36.908674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf" event={"ID":"bbee3c66-965d-4f2a-9432-a090d34ca64a","Type":"ContainerDied","Data":"c90fde3c50a9f2ea99fb48ebcda8044087bd0c2346389b3014cc45fcac67c639"}
Jul 17 00:20:38.031411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.031387    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:38.149854 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.149818    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwqrr\" (UniqueName: \"kubernetes.io/projected/bbee3c66-965d-4f2a-9432-a090d34ca64a-kube-api-access-qwqrr\") pod \"bbee3c66-965d-4f2a-9432-a090d34ca64a\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") "
Jul 17 00:20:38.149989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.149892    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-util\") pod \"bbee3c66-965d-4f2a-9432-a090d34ca64a\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") "
Jul 17 00:20:38.149989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.149942    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-bundle\") pod \"bbee3c66-965d-4f2a-9432-a090d34ca64a\" (UID: \"bbee3c66-965d-4f2a-9432-a090d34ca64a\") "
Jul 17 00:20:38.150706 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.150673    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-bundle" (OuterVolumeSpecName: "bundle") pod "bbee3c66-965d-4f2a-9432-a090d34ca64a" (UID: "bbee3c66-965d-4f2a-9432-a090d34ca64a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:20:38.151908 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.151887    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbee3c66-965d-4f2a-9432-a090d34ca64a-kube-api-access-qwqrr" (OuterVolumeSpecName: "kube-api-access-qwqrr") pod "bbee3c66-965d-4f2a-9432-a090d34ca64a" (UID: "bbee3c66-965d-4f2a-9432-a090d34ca64a"). InnerVolumeSpecName "kube-api-access-qwqrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:20:38.154500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.154479    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-util" (OuterVolumeSpecName: "util") pod "bbee3c66-965d-4f2a-9432-a090d34ca64a" (UID: "bbee3c66-965d-4f2a-9432-a090d34ca64a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:20:38.250588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.250537    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-util\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:20:38.250588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.250560    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbee3c66-965d-4f2a-9432-a090d34ca64a-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:20:38.250588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.250569    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwqrr\" (UniqueName: \"kubernetes.io/projected/bbee3c66-965d-4f2a-9432-a090d34ca64a-kube-api-access-qwqrr\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:20:38.917045 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.917009    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf" event={"ID":"bbee3c66-965d-4f2a-9432-a090d34ca64a","Type":"ContainerDied","Data":"c2d7a52ca7551075f74f1f5af1248ae81844d429af8c62568fb096f4370dfcf6"}
Jul 17 00:20:38.917198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.917056    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zx5cf"
Jul 17 00:20:38.917198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:38.917068    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2d7a52ca7551075f74f1f5af1248ae81844d429af8c62568fb096f4370dfcf6"
Jul 17 00:20:45.419733 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419586    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"]
Jul 17 00:20:45.420221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419915    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerName="util"
Jul 17 00:20:45.420221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419928    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerName="util"
Jul 17 00:20:45.420221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419936    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerName="extract"
Jul 17 00:20:45.420221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419941    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerName="extract"
Jul 17 00:20:45.420221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419950    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerName="pull"
Jul 17 00:20:45.420221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419955    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerName="pull"
Jul 17 00:20:45.420221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.419991    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbee3c66-965d-4f2a-9432-a090d34ca64a" containerName="extract"
Jul 17 00:20:45.423192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.423174    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.425246 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.425227    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 17 00:20:45.425702 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.425682    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 17 00:20:45.425977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.425961    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-smcr8\""
Jul 17 00:20:45.435294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.435274    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"]
Jul 17 00:20:45.496053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.496026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8shf\" (UniqueName: \"kubernetes.io/projected/9212c2f3-021b-4932-b1a3-b785afc8a8c7-kube-api-access-j8shf\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8546\" (UID: \"9212c2f3-021b-4932-b1a3-b785afc8a8c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.496150 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.496074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9212c2f3-021b-4932-b1a3-b785afc8a8c7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8546\" (UID: \"9212c2f3-021b-4932-b1a3-b785afc8a8c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.596325 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.596303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8shf\" (UniqueName: \"kubernetes.io/projected/9212c2f3-021b-4932-b1a3-b785afc8a8c7-kube-api-access-j8shf\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8546\" (UID: \"9212c2f3-021b-4932-b1a3-b785afc8a8c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.596397 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.596344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9212c2f3-021b-4932-b1a3-b785afc8a8c7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8546\" (UID: \"9212c2f3-021b-4932-b1a3-b785afc8a8c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.596632 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.596617    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9212c2f3-021b-4932-b1a3-b785afc8a8c7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8546\" (UID: \"9212c2f3-021b-4932-b1a3-b785afc8a8c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.607190 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.607164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8shf\" (UniqueName: \"kubernetes.io/projected/9212c2f3-021b-4932-b1a3-b785afc8a8c7-kube-api-access-j8shf\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8546\" (UID: \"9212c2f3-021b-4932-b1a3-b785afc8a8c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.732038 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.731981    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"
Jul 17 00:20:45.862581 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.862548    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546"]
Jul 17 00:20:45.865866 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:20:45.865840    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9212c2f3_021b_4932_b1a3_b785afc8a8c7.slice/crio-ff3d90a6877df813b9bfb9275f449cf306d99eb29eb189dec5a500ef6abb46b4 WatchSource:0}: Error finding container ff3d90a6877df813b9bfb9275f449cf306d99eb29eb189dec5a500ef6abb46b4: Status 404 returned error can't find the container with id ff3d90a6877df813b9bfb9275f449cf306d99eb29eb189dec5a500ef6abb46b4
Jul 17 00:20:45.942386 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:45.942358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546" event={"ID":"9212c2f3-021b-4932-b1a3-b785afc8a8c7","Type":"ContainerStarted","Data":"ff3d90a6877df813b9bfb9275f449cf306d99eb29eb189dec5a500ef6abb46b4"}
Jul 17 00:20:47.952719 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:47.952640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546" event={"ID":"9212c2f3-021b-4932-b1a3-b785afc8a8c7","Type":"ContainerStarted","Data":"4c58e57369b1fcd3e46c80d94592e749953c5aa40535e6d520b01046462753bb"}
Jul 17 00:20:47.971081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:47.971027    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8546" podStartSLOduration=1.267337259 podStartE2EDuration="2.971008317s" totalImagesPullingTime="1.703671058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:20:47.969602501 +0000 UTC m=+242.545755075" watchObservedRunningTime="2026-07-17 00:20:47.971008317 +0000 UTC m=+242.547160892"
Jul 17 00:20:48.855417 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:48.855383    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n7b4z"]
Jul 17 00:20:48.857329 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:48.857313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:48.859261 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:48.859243    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 00:20:48.859662 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:48.859644    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 00:20:48.859662 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:48.859662    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vv2zs\""
Jul 17 00:20:48.868150 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:48.868129    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n7b4z"]
Jul 17 00:20:49.020688 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.020665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e953edab-7dac-4ade-89dc-c91d5faf92b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n7b4z\" (UID: \"e953edab-7dac-4ade-89dc-c91d5faf92b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:49.021008 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.020716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spzm7\" (UniqueName: \"kubernetes.io/projected/e953edab-7dac-4ade-89dc-c91d5faf92b9-kube-api-access-spzm7\") pod \"cert-manager-webhook-6486ffd997-n7b4z\" (UID: \"e953edab-7dac-4ade-89dc-c91d5faf92b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:49.121924 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.121875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzm7\" (UniqueName: \"kubernetes.io/projected/e953edab-7dac-4ade-89dc-c91d5faf92b9-kube-api-access-spzm7\") pod \"cert-manager-webhook-6486ffd997-n7b4z\" (UID: \"e953edab-7dac-4ade-89dc-c91d5faf92b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:49.122007 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.121926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e953edab-7dac-4ade-89dc-c91d5faf92b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n7b4z\" (UID: \"e953edab-7dac-4ade-89dc-c91d5faf92b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:49.129543 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.129517    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e953edab-7dac-4ade-89dc-c91d5faf92b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n7b4z\" (UID: \"e953edab-7dac-4ade-89dc-c91d5faf92b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:49.129902 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.129885    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzm7\" (UniqueName: \"kubernetes.io/projected/e953edab-7dac-4ade-89dc-c91d5faf92b9-kube-api-access-spzm7\") pod \"cert-manager-webhook-6486ffd997-n7b4z\" (UID: \"e953edab-7dac-4ade-89dc-c91d5faf92b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:49.179672 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.179649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:20:49.308823 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.308784    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n7b4z"]
Jul 17 00:20:49.318020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.317909    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"]
Jul 17 00:20:49.321065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.321048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.323149 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.323125    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nt95l\""
Jul 17 00:20:49.323486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.323464    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 00:20:49.323601 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.323510    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 00:20:49.324274 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.324250    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"]
Jul 17 00:20:49.424283 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.424228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.424385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.424278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crp5s\" (UniqueName: \"kubernetes.io/projected/9e90ea27-e09a-466e-be4f-777b230e64f5-kube-api-access-crp5s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.424385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.424307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.525260 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.525239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.525355 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.525288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.525355 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.525319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crp5s\" (UniqueName: \"kubernetes.io/projected/9e90ea27-e09a-466e-be4f-777b230e64f5-kube-api-access-crp5s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.525615 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.525596    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.525679 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.525639    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.533376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.533355    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crp5s\" (UniqueName: \"kubernetes.io/projected/9e90ea27-e09a-466e-be4f-777b230e64f5-kube-api-access-crp5s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.631485 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.631463    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:49.755539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.755516    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"]
Jul 17 00:20:49.757710 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:20:49.757684    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e90ea27_e09a_466e_be4f_777b230e64f5.slice/crio-e17c9091dd9b5f548897339af48370ad20a71b5341c33ea0ca792e1e7186626a WatchSource:0}: Error finding container e17c9091dd9b5f548897339af48370ad20a71b5341c33ea0ca792e1e7186626a: Status 404 returned error can't find the container with id e17c9091dd9b5f548897339af48370ad20a71b5341c33ea0ca792e1e7186626a
Jul 17 00:20:49.961495 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.961437    2573 generic.go:356] "Generic (PLEG): container finished" podID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerID="f280ea9e26d097f6103fd2eb79b87648e3693096b522e28d42ead9009b1c4e09" exitCode=0
Jul 17 00:20:49.961617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.961507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt" event={"ID":"9e90ea27-e09a-466e-be4f-777b230e64f5","Type":"ContainerDied","Data":"f280ea9e26d097f6103fd2eb79b87648e3693096b522e28d42ead9009b1c4e09"}
Jul 17 00:20:49.961617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.961545    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt" event={"ID":"9e90ea27-e09a-466e-be4f-777b230e64f5","Type":"ContainerStarted","Data":"e17c9091dd9b5f548897339af48370ad20a71b5341c33ea0ca792e1e7186626a"}
Jul 17 00:20:49.967254 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:49.967232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z" event={"ID":"e953edab-7dac-4ade-89dc-c91d5faf92b9","Type":"ContainerStarted","Data":"7faa00e8987f991d9a6be66725b0ddeedf7fb0b56a5349319309b15893c0e406"}
Jul 17 00:20:52.976627 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:52.976599    2573 generic.go:356] "Generic (PLEG): container finished" podID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerID="88244b9cdfaf0c17794bf4f5cc473b4e71a224b133a14666dd455ea2db91b32e" exitCode=0
Jul 17 00:20:52.977040 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:52.976678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt" event={"ID":"9e90ea27-e09a-466e-be4f-777b230e64f5","Type":"ContainerDied","Data":"88244b9cdfaf0c17794bf4f5cc473b4e71a224b133a14666dd455ea2db91b32e"}
Jul 17 00:20:53.981592 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:53.981557    2573 generic.go:356] "Generic (PLEG): container finished" podID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerID="032ddf2f5034e95384a436993186aed88687eb066e52f6f52fe5c0b7ace81754" exitCode=0
Jul 17 00:20:53.981592 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:53.981595    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt" event={"ID":"9e90ea27-e09a-466e-be4f-777b230e64f5","Type":"ContainerDied","Data":"032ddf2f5034e95384a436993186aed88687eb066e52f6f52fe5c0b7ace81754"}
Jul 17 00:20:55.106862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.106840    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:55.266487 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.266461    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crp5s\" (UniqueName: \"kubernetes.io/projected/9e90ea27-e09a-466e-be4f-777b230e64f5-kube-api-access-crp5s\") pod \"9e90ea27-e09a-466e-be4f-777b230e64f5\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") "
Jul 17 00:20:55.266637 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.266515    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-bundle\") pod \"9e90ea27-e09a-466e-be4f-777b230e64f5\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") "
Jul 17 00:20:55.266637 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.266552    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-util\") pod \"9e90ea27-e09a-466e-be4f-777b230e64f5\" (UID: \"9e90ea27-e09a-466e-be4f-777b230e64f5\") "
Jul 17 00:20:55.266952 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.266927    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-bundle" (OuterVolumeSpecName: "bundle") pod "9e90ea27-e09a-466e-be4f-777b230e64f5" (UID: "9e90ea27-e09a-466e-be4f-777b230e64f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:20:55.268553 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.268534    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e90ea27-e09a-466e-be4f-777b230e64f5-kube-api-access-crp5s" (OuterVolumeSpecName: "kube-api-access-crp5s") pod "9e90ea27-e09a-466e-be4f-777b230e64f5" (UID: "9e90ea27-e09a-466e-be4f-777b230e64f5"). InnerVolumeSpecName "kube-api-access-crp5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:20:55.271231 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.271186    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-util" (OuterVolumeSpecName: "util") pod "9e90ea27-e09a-466e-be4f-777b230e64f5" (UID: "9e90ea27-e09a-466e-be4f-777b230e64f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:20:55.367473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.367441    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:20:55.367473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.367466    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e90ea27-e09a-466e-be4f-777b230e64f5-util\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:20:55.367473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.367476    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crp5s\" (UniqueName: \"kubernetes.io/projected/9e90ea27-e09a-466e-be4f-777b230e64f5-kube-api-access-crp5s\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:20:55.990267 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.990227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt" event={"ID":"9e90ea27-e09a-466e-be4f-777b230e64f5","Type":"ContainerDied","Data":"e17c9091dd9b5f548897339af48370ad20a71b5341c33ea0ca792e1e7186626a"}
Jul 17 00:20:55.990267 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.990269    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcnwrt"
Jul 17 00:20:55.990422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:55.990279    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e17c9091dd9b5f548897339af48370ad20a71b5341c33ea0ca792e1e7186626a"
Jul 17 00:20:59.888546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888511    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mnmq2"]
Jul 17 00:20:59.888957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888775    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerName="util"
Jul 17 00:20:59.888957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888786    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerName="util"
Jul 17 00:20:59.888957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888816    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerName="extract"
Jul 17 00:20:59.888957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888825    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerName="extract"
Jul 17 00:20:59.888957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888835    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerName="pull"
Jul 17 00:20:59.888957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888840    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerName="pull"
Jul 17 00:20:59.888957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.888908    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e90ea27-e09a-466e-be4f-777b230e64f5" containerName="extract"
Jul 17 00:20:59.892724 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.892707    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:20:59.894557 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.894537    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-glrs8\""
Jul 17 00:20:59.901576 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.900927    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mnmq2"]
Jul 17 00:20:59.994095 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.994061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77e3dc73-1049-4d17-8a46-06d483b5cedc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mnmq2\" (UID: \"77e3dc73-1049-4d17-8a46-06d483b5cedc\") " pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:20:59.994095 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:20:59.994096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6sr6\" (UniqueName: \"kubernetes.io/projected/77e3dc73-1049-4d17-8a46-06d483b5cedc-kube-api-access-r6sr6\") pod \"cert-manager-55b4d954d5-mnmq2\" (UID: \"77e3dc73-1049-4d17-8a46-06d483b5cedc\") " pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:21:00.095371 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:00.095340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77e3dc73-1049-4d17-8a46-06d483b5cedc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mnmq2\" (UID: \"77e3dc73-1049-4d17-8a46-06d483b5cedc\") " pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:21:00.095371 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:00.095374    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6sr6\" (UniqueName: \"kubernetes.io/projected/77e3dc73-1049-4d17-8a46-06d483b5cedc-kube-api-access-r6sr6\") pod \"cert-manager-55b4d954d5-mnmq2\" (UID: \"77e3dc73-1049-4d17-8a46-06d483b5cedc\") " pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:21:00.102156 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:00.102131    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77e3dc73-1049-4d17-8a46-06d483b5cedc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mnmq2\" (UID: \"77e3dc73-1049-4d17-8a46-06d483b5cedc\") " pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:21:00.102271 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:00.102256    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6sr6\" (UniqueName: \"kubernetes.io/projected/77e3dc73-1049-4d17-8a46-06d483b5cedc-kube-api-access-r6sr6\") pod \"cert-manager-55b4d954d5-mnmq2\" (UID: \"77e3dc73-1049-4d17-8a46-06d483b5cedc\") " pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:21:00.206002 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:00.205922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mnmq2"
Jul 17 00:21:00.327832 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:00.327780    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mnmq2"]
Jul 17 00:21:00.331108 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:00.331075    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77e3dc73_1049_4d17_8a46_06d483b5cedc.slice/crio-1ac68737f1c4a3bae2f143472b97d14e762414c6a69132e4928b7a401e4c91bb WatchSource:0}: Error finding container 1ac68737f1c4a3bae2f143472b97d14e762414c6a69132e4928b7a401e4c91bb: Status 404 returned error can't find the container with id 1ac68737f1c4a3bae2f143472b97d14e762414c6a69132e4928b7a401e4c91bb
Jul 17 00:21:01.011058 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:01.011024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mnmq2" event={"ID":"77e3dc73-1049-4d17-8a46-06d483b5cedc","Type":"ContainerStarted","Data":"1ac68737f1c4a3bae2f143472b97d14e762414c6a69132e4928b7a401e4c91bb"}
Jul 17 00:21:04.024925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:04.024892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mnmq2" event={"ID":"77e3dc73-1049-4d17-8a46-06d483b5cedc","Type":"ContainerStarted","Data":"fc18b675809c7d9e827e042e422c60236eea3305763f194418429bcf7138c1c2"}
Jul 17 00:21:04.026165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:04.026142    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z" event={"ID":"e953edab-7dac-4ade-89dc-c91d5faf92b9","Type":"ContainerStarted","Data":"e704507b3e4614555df117e39c92a271ba1f5e4e4dbc4fa1fb21061810dec929"}
Jul 17 00:21:04.026296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:04.026277    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:21:04.038940 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:04.038897    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mnmq2" podStartSLOduration=2.111028945 podStartE2EDuration="5.038884264s" totalImagesPullingTime="2.927855319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:04.038319296 +0000 UTC m=+258.614471866" watchObservedRunningTime="2026-07-17 00:21:04.038884264 +0000 UTC m=+258.615036838"
Jul 17 00:21:04.055445 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:04.055400    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z" podStartSLOduration=2.105981524 podStartE2EDuration="16.055388046s" totalImagesPullingTime="13.949406522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:04.053826969 +0000 UTC m=+258.629979544" watchObservedRunningTime="2026-07-17 00:21:04.055388046 +0000 UTC m=+258.631540622"
Jul 17 00:21:09.904107 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:09.904033    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"]
Jul 17 00:21:09.919968 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:09.919943    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"]
Jul 17 00:21:09.920116 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:09.920035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:09.921973 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:09.921948    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 00:21:09.922455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:09.922433    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 00:21:09.922570 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:09.922433    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nt95l\""
Jul 17 00:21:10.037524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.037494    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n7b4z"
Jul 17 00:21:10.064126 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.064102    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.064281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.064156    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxjhf\" (UniqueName: \"kubernetes.io/projected/6119022b-5f4a-4c1c-a058-816a9c66f367-kube-api-access-fxjhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.064281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.064239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.165438 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.165367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.165573 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.165436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxjhf\" (UniqueName: \"kubernetes.io/projected/6119022b-5f4a-4c1c-a058-816a9c66f367-kube-api-access-fxjhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.165573 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.165493    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.165824 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.165777    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.165889 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.165826    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.173243 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.173218    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxjhf\" (UniqueName: \"kubernetes.io/projected/6119022b-5f4a-4c1c-a058-816a9c66f367-kube-api-access-fxjhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.229379 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.229354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:10.369383 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:10.369352    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"]
Jul 17 00:21:10.372513 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:10.372477    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6119022b_5f4a_4c1c_a058_816a9c66f367.slice/crio-487b2bfd624c25d28cb4b11d3a5f16930e2dfa049894dcac330cc3f50b1a0b57 WatchSource:0}: Error finding container 487b2bfd624c25d28cb4b11d3a5f16930e2dfa049894dcac330cc3f50b1a0b57: Status 404 returned error can't find the container with id 487b2bfd624c25d28cb4b11d3a5f16930e2dfa049894dcac330cc3f50b1a0b57
Jul 17 00:21:11.058185 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:11.058149    2573 generic.go:356] "Generic (PLEG): container finished" podID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerID="32fbb942a3434348f5b4c63c1ff67a0e2580abe59d2377ebaf8d8e1db6d61e1b" exitCode=0
Jul 17 00:21:11.058664 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:11.058270    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5" event={"ID":"6119022b-5f4a-4c1c-a058-816a9c66f367","Type":"ContainerDied","Data":"32fbb942a3434348f5b4c63c1ff67a0e2580abe59d2377ebaf8d8e1db6d61e1b"}
Jul 17 00:21:11.058664 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:11.058362    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5" event={"ID":"6119022b-5f4a-4c1c-a058-816a9c66f367","Type":"ContainerStarted","Data":"487b2bfd624c25d28cb4b11d3a5f16930e2dfa049894dcac330cc3f50b1a0b57"}
Jul 17 00:21:12.063672 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:12.063639    2573 generic.go:356] "Generic (PLEG): container finished" podID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerID="919f87955e5dddf9c77e3c6c751542beefd06af91a40fd570cb144987ea07d74" exitCode=0
Jul 17 00:21:12.064048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:12.063711    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5" event={"ID":"6119022b-5f4a-4c1c-a058-816a9c66f367","Type":"ContainerDied","Data":"919f87955e5dddf9c77e3c6c751542beefd06af91a40fd570cb144987ea07d74"}
Jul 17 00:21:13.069553 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:13.069523    2573 generic.go:356] "Generic (PLEG): container finished" podID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerID="a6e316f647a46e8cc206f9aa56d14322f4920e649f62a6ebf7fe800118568661" exitCode=0
Jul 17 00:21:13.069937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:13.069609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5" event={"ID":"6119022b-5f4a-4c1c-a058-816a9c66f367","Type":"ContainerDied","Data":"a6e316f647a46e8cc206f9aa56d14322f4920e649f62a6ebf7fe800118568661"}
Jul 17 00:21:14.190303 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.190279    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:14.293469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.293448    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-bundle\") pod \"6119022b-5f4a-4c1c-a058-816a9c66f367\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") "
Jul 17 00:21:14.293579 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.293492    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-util\") pod \"6119022b-5f4a-4c1c-a058-816a9c66f367\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") "
Jul 17 00:21:14.293579 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.293513    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxjhf\" (UniqueName: \"kubernetes.io/projected/6119022b-5f4a-4c1c-a058-816a9c66f367-kube-api-access-fxjhf\") pod \"6119022b-5f4a-4c1c-a058-816a9c66f367\" (UID: \"6119022b-5f4a-4c1c-a058-816a9c66f367\") "
Jul 17 00:21:14.294132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.294110    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-bundle" (OuterVolumeSpecName: "bundle") pod "6119022b-5f4a-4c1c-a058-816a9c66f367" (UID: "6119022b-5f4a-4c1c-a058-816a9c66f367"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:21:14.295513 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.295486    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6119022b-5f4a-4c1c-a058-816a9c66f367-kube-api-access-fxjhf" (OuterVolumeSpecName: "kube-api-access-fxjhf") pod "6119022b-5f4a-4c1c-a058-816a9c66f367" (UID: "6119022b-5f4a-4c1c-a058-816a9c66f367"). InnerVolumeSpecName "kube-api-access-fxjhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:21:14.299996 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.299970    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-util" (OuterVolumeSpecName: "util") pod "6119022b-5f4a-4c1c-a058-816a9c66f367" (UID: "6119022b-5f4a-4c1c-a058-816a9c66f367"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:21:14.394146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.394099    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:21:14.394146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.394118    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6119022b-5f4a-4c1c-a058-816a9c66f367-util\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:21:14.394146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:14.394127    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxjhf\" (UniqueName: \"kubernetes.io/projected/6119022b-5f4a-4c1c-a058-816a9c66f367-kube-api-access-fxjhf\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:21:15.079702 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:15.079673    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5"
Jul 17 00:21:15.079702 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:15.079686    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj72m5" event={"ID":"6119022b-5f4a-4c1c-a058-816a9c66f367","Type":"ContainerDied","Data":"487b2bfd624c25d28cb4b11d3a5f16930e2dfa049894dcac330cc3f50b1a0b57"}
Jul 17 00:21:15.079913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:15.079727    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="487b2bfd624c25d28cb4b11d3a5f16930e2dfa049894dcac330cc3f50b1a0b57"
Jul 17 00:21:23.158530 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158497    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"]
Jul 17 00:21:23.159019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158759    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerName="util"
Jul 17 00:21:23.159019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158770    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerName="util"
Jul 17 00:21:23.159019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158781    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerName="extract"
Jul 17 00:21:23.159019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158802    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerName="extract"
Jul 17 00:21:23.159019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158813    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerName="pull"
Jul 17 00:21:23.159019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158819    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerName="pull"
Jul 17 00:21:23.159019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.158859    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6119022b-5f4a-4c1c-a058-816a9c66f367" containerName="extract"
Jul 17 00:21:23.164300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.164285    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.166539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.166514    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 00:21:23.166669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.166540    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 00:21:23.166669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.166563    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 00:21:23.166669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.166574    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 00:21:23.166669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.166595    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g6jft\""
Jul 17 00:21:23.175406 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.175383    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"]
Jul 17 00:21:23.351678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.351650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-tmp\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.351845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.351685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmjr\" (UniqueName: \"kubernetes.io/projected/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-kube-api-access-prmjr\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.351845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.351706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-manifests\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.351845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.351729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.351845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.351761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.452422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.452356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-tmp\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.452422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.452394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmjr\" (UniqueName: \"kubernetes.io/projected/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-kube-api-access-prmjr\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.452422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.452414    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-manifests\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.452609 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.452437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.452609 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.452458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.452869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.452851    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-manifests\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.454814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.454769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-tmp\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.454937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.454871    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.454998 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.454949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.460300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.460278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmjr\" (UniqueName: \"kubernetes.io/projected/47849ce6-96d2-4b16-9ce6-9b6765a6b5ed-kube-api-access-prmjr\") pod \"opendatahub-operator-controller-manager-c9c65dfff-8mc6b\" (UID: \"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.474149 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.474133    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:23.601010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:23.600980    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"]
Jul 17 00:21:23.603959 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:23.603924    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47849ce6_96d2_4b16_9ce6_9b6765a6b5ed.slice/crio-678d603b6a74f883136ae6764c850b856622c00d1cd02be756d5cd9bfa8c04c1 WatchSource:0}: Error finding container 678d603b6a74f883136ae6764c850b856622c00d1cd02be756d5cd9bfa8c04c1: Status 404 returned error can't find the container with id 678d603b6a74f883136ae6764c850b856622c00d1cd02be756d5cd9bfa8c04c1
Jul 17 00:21:24.115941 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:24.115907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b" event={"ID":"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed","Type":"ContainerStarted","Data":"678d603b6a74f883136ae6764c850b856622c00d1cd02be756d5cd9bfa8c04c1"}
Jul 17 00:21:27.133104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:27.133072    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b" event={"ID":"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed","Type":"ContainerStarted","Data":"294e8f6567b7220d2de7f44eb9ab31ad9977c3370f97c9d845776abd9ebe2410"}
Jul 17 00:21:28.138958 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:28.138924    2573 generic.go:356] "Generic (PLEG): container finished" podID="47849ce6-96d2-4b16-9ce6-9b6765a6b5ed" containerID="294e8f6567b7220d2de7f44eb9ab31ad9977c3370f97c9d845776abd9ebe2410" exitCode=0
Jul 17 00:21:28.139308 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:28.139005    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b" event={"ID":"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed","Type":"ContainerDied","Data":"294e8f6567b7220d2de7f44eb9ab31ad9977c3370f97c9d845776abd9ebe2410"}
Jul 17 00:21:29.145844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:29.145780    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b" event={"ID":"47849ce6-96d2-4b16-9ce6-9b6765a6b5ed","Type":"ContainerStarted","Data":"18739fee733f6d2b6ea895da7875034da626121e1b0939c8a4c343abd9e33658"}
Jul 17 00:21:29.146275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:29.145960    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:29.167616 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:29.167573    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b" podStartSLOduration=1.831155078 podStartE2EDuration="6.167559861s" totalImagesPullingTime="3.303706425s" totalInitContainerRuntime="1.032698358s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:29.165432176 +0000 UTC m=+283.741584751" watchObservedRunningTime="2026-07-17 00:21:29.167559861 +0000 UTC m=+283.743712435"
Jul 17 00:21:40.151673 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:40.151638    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-8mc6b"
Jul 17 00:21:42.573698 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.573655    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d679d599d-75nvg"]
Jul 17 00:21:42.577847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.577825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.580114 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.580083    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 00:21:42.580281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.580117    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 00:21:42.580281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.580191    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 00:21:42.580865 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.580844    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-68ctg\""
Jul 17 00:21:42.581217 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.581186    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 00:21:42.590733 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.590708    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzbff\" (UniqueName: \"kubernetes.io/projected/85e79f05-6be2-4c22-b098-89eb2450e25c-kube-api-access-tzbff\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.590865 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.590777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e79f05-6be2-4c22-b098-89eb2450e25c-tls-certs\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.590865 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.590839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85e79f05-6be2-4c22-b098-89eb2450e25c-tmp\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.592089 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.592069    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d679d599d-75nvg"]
Jul 17 00:21:42.651283 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.651248    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v92p8"]
Jul 17 00:21:42.655734 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.655709    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:42.663855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.663836    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-txfrs\""
Jul 17 00:21:42.664819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.664780    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 17 00:21:42.685228 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.685207    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v92p8"]
Jul 17 00:21:42.691694 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.691660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e79f05-6be2-4c22-b098-89eb2450e25c-tls-certs\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.691816 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.691721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxg6b\" (UniqueName: \"kubernetes.io/projected/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-kube-api-access-sxg6b\") pod \"odh-model-controller-7998c67dc4-v92p8\" (UID: \"8b2d5d3c-64f4-4d5c-a629-60819a1262a6\") " pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:42.691816 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.691754    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85e79f05-6be2-4c22-b098-89eb2450e25c-tmp\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.691939 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.691867    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-cert\") pod \"odh-model-controller-7998c67dc4-v92p8\" (UID: \"8b2d5d3c-64f4-4d5c-a629-60819a1262a6\") " pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:42.691991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.691934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzbff\" (UniqueName: \"kubernetes.io/projected/85e79f05-6be2-4c22-b098-89eb2450e25c-kube-api-access-tzbff\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.694547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.694523    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85e79f05-6be2-4c22-b098-89eb2450e25c-tmp\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.694721 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.694705    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e79f05-6be2-4c22-b098-89eb2450e25c-tls-certs\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.709498 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.709476    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzbff\" (UniqueName: \"kubernetes.io/projected/85e79f05-6be2-4c22-b098-89eb2450e25c-kube-api-access-tzbff\") pod \"kube-auth-proxy-d679d599d-75nvg\" (UID: \"85e79f05-6be2-4c22-b098-89eb2450e25c\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:42.793284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.793247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxg6b\" (UniqueName: \"kubernetes.io/projected/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-kube-api-access-sxg6b\") pod \"odh-model-controller-7998c67dc4-v92p8\" (UID: \"8b2d5d3c-64f4-4d5c-a629-60819a1262a6\") " pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:42.793453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.793297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-cert\") pod \"odh-model-controller-7998c67dc4-v92p8\" (UID: \"8b2d5d3c-64f4-4d5c-a629-60819a1262a6\") " pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:42.793453 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:21:42.793421    2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 17 00:21:42.793568 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:21:42.793477    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-cert podName:8b2d5d3c-64f4-4d5c-a629-60819a1262a6 nodeName:}" failed. No retries permitted until 2026-07-17 00:21:43.293460518 +0000 UTC m=+297.869613073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-cert") pod "odh-model-controller-7998c67dc4-v92p8" (UID: "8b2d5d3c-64f4-4d5c-a629-60819a1262a6") : secret "odh-model-controller-webhook-cert" not found
Jul 17 00:21:42.830360 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.830293    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxg6b\" (UniqueName: \"kubernetes.io/projected/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-kube-api-access-sxg6b\") pod \"odh-model-controller-7998c67dc4-v92p8\" (UID: \"8b2d5d3c-64f4-4d5c-a629-60819a1262a6\") " pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:42.888411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:42.888368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg"
Jul 17 00:21:43.027655 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.027476    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d679d599d-75nvg"]
Jul 17 00:21:43.030362 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:43.030329    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e79f05_6be2_4c22_b098_89eb2450e25c.slice/crio-e6f2f5d766a9df77f5c119a873538d42d54285b5ec8b6d6a33cd4cb0cfc28d9a WatchSource:0}: Error finding container e6f2f5d766a9df77f5c119a873538d42d54285b5ec8b6d6a33cd4cb0cfc28d9a: Status 404 returned error can't find the container with id e6f2f5d766a9df77f5c119a873538d42d54285b5ec8b6d6a33cd4cb0cfc28d9a
Jul 17 00:21:43.201775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.201696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg" event={"ID":"85e79f05-6be2-4c22-b098-89eb2450e25c","Type":"ContainerStarted","Data":"e6f2f5d766a9df77f5c119a873538d42d54285b5ec8b6d6a33cd4cb0cfc28d9a"}
Jul 17 00:21:43.296880 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.296845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-cert\") pod \"odh-model-controller-7998c67dc4-v92p8\" (UID: \"8b2d5d3c-64f4-4d5c-a629-60819a1262a6\") " pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:43.299317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.299293    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2d5d3c-64f4-4d5c-a629-60819a1262a6-cert\") pod \"odh-model-controller-7998c67dc4-v92p8\" (UID: \"8b2d5d3c-64f4-4d5c-a629-60819a1262a6\") " pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:43.448740 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.448702    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xvwgt"]
Jul 17 00:21:43.453906 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.453845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.456330 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.456306    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 17 00:21:43.479556 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.479530    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xvwgt"]
Jul 17 00:21:43.498323 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.498301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbpf\" (UniqueName: \"kubernetes.io/projected/7a35fd16-bf16-46a5-a26a-8e3d83d0b66b-kube-api-access-mbbpf\") pod \"model-serving-api-54d89bd5fc-xvwgt\" (UID: \"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b\") " pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.498431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.498340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a35fd16-bf16-46a5-a26a-8e3d83d0b66b-tls-certs\") pod \"model-serving-api-54d89bd5fc-xvwgt\" (UID: \"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b\") " pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.565727 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.565695    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:43.599526 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.599492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbpf\" (UniqueName: \"kubernetes.io/projected/7a35fd16-bf16-46a5-a26a-8e3d83d0b66b-kube-api-access-mbbpf\") pod \"model-serving-api-54d89bd5fc-xvwgt\" (UID: \"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b\") " pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.599852 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.599562    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a35fd16-bf16-46a5-a26a-8e3d83d0b66b-tls-certs\") pod \"model-serving-api-54d89bd5fc-xvwgt\" (UID: \"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b\") " pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.602138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.602115    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a35fd16-bf16-46a5-a26a-8e3d83d0b66b-tls-certs\") pod \"model-serving-api-54d89bd5fc-xvwgt\" (UID: \"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b\") " pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.614018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.613996    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbpf\" (UniqueName: \"kubernetes.io/projected/7a35fd16-bf16-46a5-a26a-8e3d83d0b66b-kube-api-access-mbbpf\") pod \"model-serving-api-54d89bd5fc-xvwgt\" (UID: \"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b\") " pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.718030 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.717946    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v92p8"]
Jul 17 00:21:43.720958 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:43.720930    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2d5d3c_64f4_4d5c_a629_60819a1262a6.slice/crio-ccaef2c67217c0926e02b28374faf989453e3d9213f4dcef94b12e4c28993234 WatchSource:0}: Error finding container ccaef2c67217c0926e02b28374faf989453e3d9213f4dcef94b12e4c28993234: Status 404 returned error can't find the container with id ccaef2c67217c0926e02b28374faf989453e3d9213f4dcef94b12e4c28993234
Jul 17 00:21:43.764011 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.763979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:43.926766 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:43.926729    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xvwgt"]
Jul 17 00:21:43.929260 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:43.929228    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a35fd16_bf16_46a5_a26a_8e3d83d0b66b.slice/crio-70bdf71811688816ac6f0549818c6916c51d08bfefc16ed4a8c49bc2d456d119 WatchSource:0}: Error finding container 70bdf71811688816ac6f0549818c6916c51d08bfefc16ed4a8c49bc2d456d119: Status 404 returned error can't find the container with id 70bdf71811688816ac6f0549818c6916c51d08bfefc16ed4a8c49bc2d456d119
Jul 17 00:21:44.106767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.106726    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"]
Jul 17 00:21:44.111416 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.111394    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.113856 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.113834    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 00:21:44.113950 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.113882    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8mfl9\""
Jul 17 00:21:44.113950 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.113920    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 00:21:44.114077 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.113971    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 00:21:44.114203 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.114186    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 00:21:44.123052 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.123032    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"]
Jul 17 00:21:44.205134 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.205102    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-local-certs\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.205317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.205162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.205317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.205225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.205317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.205273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.205471 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.205319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-token\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.205471 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.205361    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7tv\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-kube-api-access-mh7tv\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.205571 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.205469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-cacerts\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.207627 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.207597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt" event={"ID":"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b","Type":"ContainerStarted","Data":"70bdf71811688816ac6f0549818c6916c51d08bfefc16ed4a8c49bc2d456d119"}
Jul 17 00:21:44.208992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.208966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v92p8" event={"ID":"8b2d5d3c-64f4-4d5c-a629-60819a1262a6","Type":"ContainerStarted","Data":"ccaef2c67217c0926e02b28374faf989453e3d9213f4dcef94b12e4c28993234"}
Jul 17 00:21:44.306419 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.306391    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-token\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.306618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.306429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7tv\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-kube-api-access-mh7tv\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.306618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.306458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-cacerts\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.306618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.306480    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-local-certs\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.306618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.306504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.306618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.306531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.306618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.306551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.307201 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.307179    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.308821 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.308781    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-local-certs\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.308912 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.308834    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.309618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.309595    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-cacerts\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.309705 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.309643    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.315102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.315080    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-token\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.315597 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.315563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7tv\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-kube-api-access-mh7tv\") pod \"istiod-openshift-gateway-7574459656-spdmd\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.424026 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.423949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:44.602752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:44.600534    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"]
Jul 17 00:21:44.607949 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:44.607918    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83be3d4_68af_4bcf_8e13_f37f491f3b8c.slice/crio-6af394caa014aaf9443141a41317112587e25eac2c44ee56b9da89dc388bca67 WatchSource:0}: Error finding container 6af394caa014aaf9443141a41317112587e25eac2c44ee56b9da89dc388bca67: Status 404 returned error can't find the container with id 6af394caa014aaf9443141a41317112587e25eac2c44ee56b9da89dc388bca67
Jul 17 00:21:45.217647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:45.217595    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd" event={"ID":"d83be3d4-68af-4bcf-8e13-f37f491f3b8c","Type":"ContainerStarted","Data":"6af394caa014aaf9443141a41317112587e25eac2c44ee56b9da89dc388bca67"}
Jul 17 00:21:47.608080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:47.608046    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 00:21:47.621900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:47.621877    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:21:47.622287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:47.622270    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:21:47.631496 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:47.631474    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 00:21:48.231748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.231704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg" event={"ID":"85e79f05-6be2-4c22-b098-89eb2450e25c","Type":"ContainerStarted","Data":"8df17134125732f2b01d5e830d9979b1033cd8bded799960caa67022730f105d"}
Jul 17 00:21:48.233265 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.233234    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v92p8" event={"ID":"8b2d5d3c-64f4-4d5c-a629-60819a1262a6","Type":"ContainerStarted","Data":"14b23521de715f52bd0cd6b6eedb56ae612284eda213efa54971a3365048f4c0"}
Jul 17 00:21:48.233394 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.233364    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:48.234729 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.234702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt" event={"ID":"7a35fd16-bf16-46a5-a26a-8e3d83d0b66b","Type":"ContainerStarted","Data":"a5bbacbc0216b004302b7a217cd3a4aea58a0e9c5ed40143f1dbdb1e615fb583"}
Jul 17 00:21:48.234881 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.234840    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:21:48.246974 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.246921    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d679d599d-75nvg" podStartSLOduration=1.673258025 podStartE2EDuration="6.246904987s" totalImagesPullingTime="4.573646962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:48.245214611 +0000 UTC m=+302.821367187" watchObservedRunningTime="2026-07-17 00:21:48.246904987 +0000 UTC m=+302.823057562"
Jul 17 00:21:48.261746 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.261693    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt" podStartSLOduration=1.58640764 podStartE2EDuration="5.261674282s" totalImagesPullingTime="3.675266642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:48.258919778 +0000 UTC m=+302.835072354" watchObservedRunningTime="2026-07-17 00:21:48.261674282 +0000 UTC m=+302.837826859"
Jul 17 00:21:48.274696 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:48.274649    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-v92p8" podStartSLOduration=2.377635584 podStartE2EDuration="6.274632803s" totalImagesPullingTime="3.896997219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:48.273546478 +0000 UTC m=+302.849699053" watchObservedRunningTime="2026-07-17 00:21:48.274632803 +0000 UTC m=+302.850785377"
Jul 17 00:21:50.827048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.827015    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"]
Jul 17 00:21:50.830694 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.830669    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.841953 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.841920    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"]
Jul 17 00:21:50.867150 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.867113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e44104-5be3-4c4f-b021-e22713374ecf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.867150 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.867153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.867378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.867172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whd7j\" (UniqueName: \"kubernetes.io/projected/48e44104-5be3-4c4f-b021-e22713374ecf-kube-api-access-whd7j\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.867378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.867221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e44104-5be3-4c4f-b021-e22713374ecf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.867378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.867290    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.867378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.867314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.867378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.867360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48e44104-5be3-4c4f-b021-e22713374ecf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.968163 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.968122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e44104-5be3-4c4f-b021-e22713374ecf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.968163 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.968164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.968435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.968182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whd7j\" (UniqueName: \"kubernetes.io/projected/48e44104-5be3-4c4f-b021-e22713374ecf-kube-api-access-whd7j\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.968435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.968204    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e44104-5be3-4c4f-b021-e22713374ecf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.968435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.968394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.968591 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.968439    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.968591 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.968500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48e44104-5be3-4c4f-b021-e22713374ecf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.969095 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.969069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48e44104-5be3-4c4f-b021-e22713374ecf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.970806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.970748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.970913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.970764    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.970913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.970845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e44104-5be3-4c4f-b021-e22713374ecf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.971038 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.971015    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48e44104-5be3-4c4f-b021-e22713374ecf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.978940 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.978887    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whd7j\" (UniqueName: \"kubernetes.io/projected/48e44104-5be3-4c4f-b021-e22713374ecf-kube-api-access-whd7j\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:50.978940 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:50.978892    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e44104-5be3-4c4f-b021-e22713374ecf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jndrl\" (UID: \"48e44104-5be3-4c4f-b021-e22713374ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:51.139338 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:51.139235    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:51.306439 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:51.306407    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"]
Jul 17 00:21:51.309593 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:21:51.309558    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e44104_5be3_4c4f_b021_e22713374ecf.slice/crio-1bc21be6115e6611a6e9ab1642edf874baf6935cb81b55b6e91216dda7fc6a2c WatchSource:0}: Error finding container 1bc21be6115e6611a6e9ab1642edf874baf6935cb81b55b6e91216dda7fc6a2c: Status 404 returned error can't find the container with id 1bc21be6115e6611a6e9ab1642edf874baf6935cb81b55b6e91216dda7fc6a2c
Jul 17 00:21:51.312245 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:51.312225    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 00:21:52.256714 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:52.256566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl" event={"ID":"48e44104-5be3-4c4f-b021-e22713374ecf","Type":"ContainerStarted","Data":"1bc21be6115e6611a6e9ab1642edf874baf6935cb81b55b6e91216dda7fc6a2c"}
Jul 17 00:21:53.684606 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:53.684568    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 00:21:53.685002 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:53.684654    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 00:21:54.267266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:54.267227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl" event={"ID":"48e44104-5be3-4c4f-b021-e22713374ecf","Type":"ContainerStarted","Data":"5bff55701d138cdbb2a3d0e1bec4cad6fe5109e5bca9a70e9c22d290cc07c570"}
Jul 17 00:21:54.267572 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:54.267374    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:54.269171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:54.269145    2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-jndrl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 00:21:54.269278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:54.269197    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl" podUID="48e44104-5be3-4c4f-b021-e22713374ecf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 00:21:54.288355 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:54.288308    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl" podStartSLOduration=1.9163254269999999 podStartE2EDuration="4.288293819s" totalImagesPullingTime="2.371968392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:54.286492642 +0000 UTC m=+308.862645216" watchObservedRunningTime="2026-07-17 00:21:54.288293819 +0000 UTC m=+308.864446393"
Jul 17 00:21:55.273495 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:55.273462    2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-jndrl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 00:21:55.273896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:55.273518    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl" podUID="48e44104-5be3-4c4f-b021-e22713374ecf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 00:21:56.849727 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:56.849697    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 00:21:56.850028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:56.849814    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 00:21:57.282994 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:57.282956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd" event={"ID":"d83be3d4-68af-4bcf-8e13-f37f491f3b8c","Type":"ContainerStarted","Data":"c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344"}
Jul 17 00:21:57.283180 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:57.283098    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:57.284781 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:57.284757    2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-spdmd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 00:21:57.284918 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:57.284816    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd" podUID="d83be3d4-68af-4bcf-8e13-f37f491f3b8c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 00:21:57.300185 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:57.300134    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd" podStartSLOduration=1.061203823 podStartE2EDuration="13.300117261s" totalImagesPullingTime="12.238913438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:57.298912677 +0000 UTC m=+311.875065251" watchObservedRunningTime="2026-07-17 00:21:57.300117261 +0000 UTC m=+311.876269837"
Jul 17 00:21:58.274461 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:58.274428    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jndrl"
Jul 17 00:21:58.288045 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:58.288013    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:21:58.348968 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:58.348937    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"]
Jul 17 00:21:59.242075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:59.242028    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-v92p8"
Jul 17 00:21:59.244165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:21:59.244142    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xvwgt"
Jul 17 00:22:00.296742 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.296704    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd" podUID="d83be3d4-68af-4bcf-8e13-f37f491f3b8c" containerName="discovery" containerID="cri-o://c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344" gracePeriod=30
Jul 17 00:22:00.556700 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.556645    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:22:00.660288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660255    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh7tv\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-kube-api-access-mh7tv\") pod \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") "
Jul 17 00:22:00.660464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660311    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-dns-cert\") pod \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") "
Jul 17 00:22:00.660464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-token\") pod \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") "
Jul 17 00:22:00.660464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660367    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-kubeconfig\") pod \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") "
Jul 17 00:22:00.660464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660388    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-local-certs\") pod \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") "
Jul 17 00:22:00.660464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660404    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-cacerts\") pod \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") "
Jul 17 00:22:00.660464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660437    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-ca-configmap\") pod \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\" (UID: \"d83be3d4-68af-4bcf-8e13-f37f491f3b8c\") "
Jul 17 00:22:00.660985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.660951    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d83be3d4-68af-4bcf-8e13-f37f491f3b8c" (UID: "d83be3d4-68af-4bcf-8e13-f37f491f3b8c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:22:00.662861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.662827    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d83be3d4-68af-4bcf-8e13-f37f491f3b8c" (UID: "d83be3d4-68af-4bcf-8e13-f37f491f3b8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:22:00.662957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.662892    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d83be3d4-68af-4bcf-8e13-f37f491f3b8c" (UID: "d83be3d4-68af-4bcf-8e13-f37f491f3b8c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:22:00.663010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.662995    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-kube-api-access-mh7tv" (OuterVolumeSpecName: "kube-api-access-mh7tv") pod "d83be3d4-68af-4bcf-8e13-f37f491f3b8c" (UID: "d83be3d4-68af-4bcf-8e13-f37f491f3b8c"). InnerVolumeSpecName "kube-api-access-mh7tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:22:00.663082 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.663014    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-cacerts" (OuterVolumeSpecName: "cacerts") pod "d83be3d4-68af-4bcf-8e13-f37f491f3b8c" (UID: "d83be3d4-68af-4bcf-8e13-f37f491f3b8c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:22:00.663082 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.663056    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d83be3d4-68af-4bcf-8e13-f37f491f3b8c" (UID: "d83be3d4-68af-4bcf-8e13-f37f491f3b8c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:22:00.663308 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.663286    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-local-certs" (OuterVolumeSpecName: "local-certs") pod "d83be3d4-68af-4bcf-8e13-f37f491f3b8c" (UID: "d83be3d4-68af-4bcf-8e13-f37f491f3b8c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:22:00.761239 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.761208    2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-kubeconfig\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:00.761239 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.761236    2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-local-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:00.761239 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.761244    2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-cacerts\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:00.761458 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.761255    2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-ca-configmap\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:00.761458 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.761265    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh7tv\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-kube-api-access-mh7tv\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:00.761458 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.761275    2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-csr-dns-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:00.761458 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:00.761284    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83be3d4-68af-4bcf-8e13-f37f491f3b8c-istio-token\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:01.302503 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.302468    2573 generic.go:356] "Generic (PLEG): container finished" podID="d83be3d4-68af-4bcf-8e13-f37f491f3b8c" containerID="c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344" exitCode=0
Jul 17 00:22:01.303017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.302515    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"
Jul 17 00:22:01.303017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.302556    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd" event={"ID":"d83be3d4-68af-4bcf-8e13-f37f491f3b8c","Type":"ContainerDied","Data":"c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344"}
Jul 17 00:22:01.303017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.302614    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-spdmd" event={"ID":"d83be3d4-68af-4bcf-8e13-f37f491f3b8c","Type":"ContainerDied","Data":"6af394caa014aaf9443141a41317112587e25eac2c44ee56b9da89dc388bca67"}
Jul 17 00:22:01.303017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.302648    2573 scope.go:117] "RemoveContainer" containerID="c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344"
Jul 17 00:22:01.311163 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.311148    2573 scope.go:117] "RemoveContainer" containerID="c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344"
Jul 17 00:22:01.311431 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:22:01.311410    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344\": container with ID starting with c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344 not found: ID does not exist" containerID="c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344"
Jul 17 00:22:01.311468 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.311439    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344"} err="failed to get container status \"c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344\": rpc error: code = NotFound desc = could not find container \"c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344\": container with ID starting with c0ec8285f70d0ae5940ad7af81e305733448378632f87490d095c46a85146344 not found: ID does not exist"
Jul 17 00:22:01.340425 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.340399    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"]
Jul 17 00:22:01.358913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.358887    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-spdmd"]
Jul 17 00:22:01.967646 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:01.967608    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d83be3d4-68af-4bcf-8e13-f37f491f3b8c" path="/var/lib/kubelet/pods/d83be3d4-68af-4bcf-8e13-f37f491f3b8c/volumes"
Jul 17 00:22:43.446399 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.446320    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8lfhw"]
Jul 17 00:22:43.446784 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.446641    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83be3d4-68af-4bcf-8e13-f37f491f3b8c" containerName="discovery"
Jul 17 00:22:43.446784 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.446654    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83be3d4-68af-4bcf-8e13-f37f491f3b8c" containerName="discovery"
Jul 17 00:22:43.446784 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.446720    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d83be3d4-68af-4bcf-8e13-f37f491f3b8c" containerName="discovery"
Jul 17 00:22:43.449570 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.449555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw"
Jul 17 00:22:43.451881 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.451856    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 00:22:43.452049 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.452029    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 00:22:43.452198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.452173    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-c78kw\""
Jul 17 00:22:43.458922 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.458862    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8lfhw"]
Jul 17 00:22:43.489011 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.488982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk465\" (UniqueName: \"kubernetes.io/projected/d35b75fb-f94b-4a71-93c8-4116eebac08a-kube-api-access-qk465\") pod \"kuadrant-operator-catalog-8lfhw\" (UID: \"d35b75fb-f94b-4a71-93c8-4116eebac08a\") " pod="kuadrant-system/kuadrant-operator-catalog-8lfhw"
Jul 17 00:22:43.589717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.589691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk465\" (UniqueName: \"kubernetes.io/projected/d35b75fb-f94b-4a71-93c8-4116eebac08a-kube-api-access-qk465\") pod \"kuadrant-operator-catalog-8lfhw\" (UID: \"d35b75fb-f94b-4a71-93c8-4116eebac08a\") " pod="kuadrant-system/kuadrant-operator-catalog-8lfhw"
Jul 17 00:22:43.597164 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.597147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk465\" (UniqueName: \"kubernetes.io/projected/d35b75fb-f94b-4a71-93c8-4116eebac08a-kube-api-access-qk465\") pod \"kuadrant-operator-catalog-8lfhw\" (UID: \"d35b75fb-f94b-4a71-93c8-4116eebac08a\") " pod="kuadrant-system/kuadrant-operator-catalog-8lfhw"
Jul 17 00:22:43.761641 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.761617    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw"
Jul 17 00:22:43.814016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.813987    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8lfhw"]
Jul 17 00:22:43.883668 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:43.883642    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8lfhw"]
Jul 17 00:22:43.886199 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:22:43.886168    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35b75fb_f94b_4a71_93c8_4116eebac08a.slice/crio-8fd9732b9b0aaf8c5c98aa94dec1f4aafb8f516963629a96f3c29324c25f6024 WatchSource:0}: Error finding container 8fd9732b9b0aaf8c5c98aa94dec1f4aafb8f516963629a96f3c29324c25f6024: Status 404 returned error can't find the container with id 8fd9732b9b0aaf8c5c98aa94dec1f4aafb8f516963629a96f3c29324c25f6024
Jul 17 00:22:44.024153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.024089    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6mc2k"]
Jul 17 00:22:44.028316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.028296    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:44.033522 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.033503    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6mc2k"]
Jul 17 00:22:44.093144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.093124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8bw\" (UniqueName: \"kubernetes.io/projected/cbc87a2e-6674-4bb4-ae2e-018e6362a9a6-kube-api-access-5r8bw\") pod \"kuadrant-operator-catalog-6mc2k\" (UID: \"cbc87a2e-6674-4bb4-ae2e-018e6362a9a6\") " pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:44.194482 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.194457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8bw\" (UniqueName: \"kubernetes.io/projected/cbc87a2e-6674-4bb4-ae2e-018e6362a9a6-kube-api-access-5r8bw\") pod \"kuadrant-operator-catalog-6mc2k\" (UID: \"cbc87a2e-6674-4bb4-ae2e-018e6362a9a6\") " pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:44.201317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.201293    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8bw\" (UniqueName: \"kubernetes.io/projected/cbc87a2e-6674-4bb4-ae2e-018e6362a9a6-kube-api-access-5r8bw\") pod \"kuadrant-operator-catalog-6mc2k\" (UID: \"cbc87a2e-6674-4bb4-ae2e-018e6362a9a6\") " pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:44.338342 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.338286    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:44.468452 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.468211    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6mc2k"]
Jul 17 00:22:44.470498 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:22:44.470462    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc87a2e_6674_4bb4_ae2e_018e6362a9a6.slice/crio-8000e0a2f626edfedd59c75c43cef4aa274b959abbfae261b0fa70b1f75e367c WatchSource:0}: Error finding container 8000e0a2f626edfedd59c75c43cef4aa274b959abbfae261b0fa70b1f75e367c: Status 404 returned error can't find the container with id 8000e0a2f626edfedd59c75c43cef4aa274b959abbfae261b0fa70b1f75e367c
Jul 17 00:22:44.500384 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.500359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw" event={"ID":"d35b75fb-f94b-4a71-93c8-4116eebac08a","Type":"ContainerStarted","Data":"8fd9732b9b0aaf8c5c98aa94dec1f4aafb8f516963629a96f3c29324c25f6024"}
Jul 17 00:22:44.501425 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:44.501402    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k" event={"ID":"cbc87a2e-6674-4bb4-ae2e-018e6362a9a6","Type":"ContainerStarted","Data":"8000e0a2f626edfedd59c75c43cef4aa274b959abbfae261b0fa70b1f75e367c"}
Jul 17 00:22:46.512381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.512347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k" event={"ID":"cbc87a2e-6674-4bb4-ae2e-018e6362a9a6","Type":"ContainerStarted","Data":"1db9e0df543ed71493ba7914f29c784cfd19419c409e9b27d0dcd79909831d8b"}
Jul 17 00:22:46.513779 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.513742    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw" event={"ID":"d35b75fb-f94b-4a71-93c8-4116eebac08a","Type":"ContainerStarted","Data":"2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6"}
Jul 17 00:22:46.513921 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.513780    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw" podUID="d35b75fb-f94b-4a71-93c8-4116eebac08a" containerName="registry-server" containerID="cri-o://2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6" gracePeriod=2
Jul 17 00:22:46.525618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.525577    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k" podStartSLOduration=1.089388548 podStartE2EDuration="2.525564305s" totalImagesPullingTime="1.436175757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:22:46.524632268 +0000 UTC m=+361.100784842" watchObservedRunningTime="2026-07-17 00:22:46.525564305 +0000 UTC m=+361.101716878"
Jul 17 00:22:46.538137 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.538101    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw" podStartSLOduration=1.518296095 podStartE2EDuration="3.538086226s" totalImagesPullingTime="2.019790131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:22:46.536952864 +0000 UTC m=+361.113105438" watchObservedRunningTime="2026-07-17 00:22:46.538086226 +0000 UTC m=+361.114238799"
Jul 17 00:22:46.754904 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.754882    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw"
Jul 17 00:22:46.817578 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.817522    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk465\" (UniqueName: \"kubernetes.io/projected/d35b75fb-f94b-4a71-93c8-4116eebac08a-kube-api-access-qk465\") pod \"d35b75fb-f94b-4a71-93c8-4116eebac08a\" (UID: \"d35b75fb-f94b-4a71-93c8-4116eebac08a\") "
Jul 17 00:22:46.819509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.819481    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35b75fb-f94b-4a71-93c8-4116eebac08a-kube-api-access-qk465" (OuterVolumeSpecName: "kube-api-access-qk465") pod "d35b75fb-f94b-4a71-93c8-4116eebac08a" (UID: "d35b75fb-f94b-4a71-93c8-4116eebac08a"). InnerVolumeSpecName "kube-api-access-qk465". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:22:46.917980 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:46.917956    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk465\" (UniqueName: \"kubernetes.io/projected/d35b75fb-f94b-4a71-93c8-4116eebac08a-kube-api-access-qk465\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:22:47.518966 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.518933    2573 generic.go:356] "Generic (PLEG): container finished" podID="d35b75fb-f94b-4a71-93c8-4116eebac08a" containerID="2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6" exitCode=0
Jul 17 00:22:47.519326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.518975    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw"
Jul 17 00:22:47.519326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.519021    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw" event={"ID":"d35b75fb-f94b-4a71-93c8-4116eebac08a","Type":"ContainerDied","Data":"2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6"}
Jul 17 00:22:47.519326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.519066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8lfhw" event={"ID":"d35b75fb-f94b-4a71-93c8-4116eebac08a","Type":"ContainerDied","Data":"8fd9732b9b0aaf8c5c98aa94dec1f4aafb8f516963629a96f3c29324c25f6024"}
Jul 17 00:22:47.519326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.519090    2573 scope.go:117] "RemoveContainer" containerID="2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6"
Jul 17 00:22:47.528132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.528114    2573 scope.go:117] "RemoveContainer" containerID="2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6"
Jul 17 00:22:47.528343 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:22:47.528327    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6\": container with ID starting with 2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6 not found: ID does not exist" containerID="2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6"
Jul 17 00:22:47.528392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.528351    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6"} err="failed to get container status \"2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6\": rpc error: code = NotFound desc = could not find container \"2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6\": container with ID starting with 2ff0f38843a5c66f035599f1cccbcb34bfeb7abaa402c4add827afeda8ddc2c6 not found: ID does not exist"
Jul 17 00:22:47.539517 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.539496    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8lfhw"]
Jul 17 00:22:47.543374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.543355    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8lfhw"]
Jul 17 00:22:47.967321 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:47.967250    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d35b75fb-f94b-4a71-93c8-4116eebac08a" path="/var/lib/kubelet/pods/d35b75fb-f94b-4a71-93c8-4116eebac08a/volumes"
Jul 17 00:22:54.339234 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:54.339202    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:54.339703 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:54.339263    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:54.361038 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:54.361014    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:54.574754 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:54.574730    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6mc2k"
Jul 17 00:22:59.405901 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.405859    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"]
Jul 17 00:22:59.406490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.406282    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35b75fb-f94b-4a71-93c8-4116eebac08a" containerName="registry-server"
Jul 17 00:22:59.406490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.406303    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35b75fb-f94b-4a71-93c8-4116eebac08a" containerName="registry-server"
Jul 17 00:22:59.406490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.406410    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d35b75fb-f94b-4a71-93c8-4116eebac08a" containerName="registry-server"
Jul 17 00:22:59.411316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.411292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.413448 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.413428    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8kgg7\""
Jul 17 00:22:59.420151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.420129    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"]
Jul 17 00:22:59.510600 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.510577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.510739 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.510615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.510739 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.510692    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrp7d\" (UniqueName: \"kubernetes.io/projected/db088276-f18e-4fd4-ab03-8fb01725451e-kube-api-access-zrp7d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.611639 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.611612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.611768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.611671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrp7d\" (UniqueName: \"kubernetes.io/projected/db088276-f18e-4fd4-ab03-8fb01725451e-kube-api-access-zrp7d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.611768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.611708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.612101 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.612082    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.612101 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.612094    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.627775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.627746    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrp7d\" (UniqueName: \"kubernetes.io/projected/db088276-f18e-4fd4-ab03-8fb01725451e-kube-api-access-zrp7d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.720636 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.720573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:22:59.850589 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.850563    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"]
Jul 17 00:22:59.852763 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:22:59.852731    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb088276_f18e_4fd4_ab03_8fb01725451e.slice/crio-487713fa83ad269a0dd0770f507340cd98d986854ee4f02755f8acf4c8ba3744 WatchSource:0}: Error finding container 487713fa83ad269a0dd0770f507340cd98d986854ee4f02755f8acf4c8ba3744: Status 404 returned error can't find the container with id 487713fa83ad269a0dd0770f507340cd98d986854ee4f02755f8acf4c8ba3744
Jul 17 00:22:59.998693 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:22:59.998620    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"]
Jul 17 00:23:00.002018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.002002    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.010421 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.010395    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"]
Jul 17 00:23:00.116997 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.116965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.116997 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.117000    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpgj6\" (UniqueName: \"kubernetes.io/projected/84c1c8fb-4cef-450e-aa29-69685b2384e3-kube-api-access-jpgj6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.117190 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.117064    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.218016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.217988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.218016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.218023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpgj6\" (UniqueName: \"kubernetes.io/projected/84c1c8fb-4cef-450e-aa29-69685b2384e3-kube-api-access-jpgj6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.218237 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.218090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.218423 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.218402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.218461 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.218414    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.226219 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.226192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpgj6\" (UniqueName: \"kubernetes.io/projected/84c1c8fb-4cef-450e-aa29-69685b2384e3-kube-api-access-jpgj6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.311474 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.311446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:00.392473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.392445    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"]
Jul 17 00:23:00.397517 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.397487    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.407676 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.407616    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"]
Jul 17 00:23:00.440732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.440706    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"]
Jul 17 00:23:00.443148 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:00.443115    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c1c8fb_4cef_450e_aa29_69685b2384e3.slice/crio-821816f09b6fd11ee3568210edf6937ad5ba1b0584d0d74c77c566a6aa5b3c54 WatchSource:0}: Error finding container 821816f09b6fd11ee3568210edf6937ad5ba1b0584d0d74c77c566a6aa5b3c54: Status 404 returned error can't find the container with id 821816f09b6fd11ee3568210edf6937ad5ba1b0584d0d74c77c566a6aa5b3c54
Jul 17 00:23:00.520694 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.520667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.520858 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.520709    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.520858 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.520740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjdj\" (UniqueName: \"kubernetes.io/projected/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-kube-api-access-6xjdj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.583481 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.583397    2573 generic.go:356] "Generic (PLEG): container finished" podID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerID="7630514fa87903ac01d68b4971b7e598b0e849a65615de18c44bfb0faa6705dc" exitCode=0
Jul 17 00:23:00.583481 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.583463    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk" event={"ID":"84c1c8fb-4cef-450e-aa29-69685b2384e3","Type":"ContainerDied","Data":"7630514fa87903ac01d68b4971b7e598b0e849a65615de18c44bfb0faa6705dc"}
Jul 17 00:23:00.583659 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.583525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk" event={"ID":"84c1c8fb-4cef-450e-aa29-69685b2384e3","Type":"ContainerStarted","Data":"821816f09b6fd11ee3568210edf6937ad5ba1b0584d0d74c77c566a6aa5b3c54"}
Jul 17 00:23:00.585076 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.585050    2573 generic.go:356] "Generic (PLEG): container finished" podID="db088276-f18e-4fd4-ab03-8fb01725451e" containerID="0ab9810899e62b9eb4738ce2a7f95cfc0d05d4746613910923e0c6800868d303" exitCode=0
Jul 17 00:23:00.585176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.585107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc" event={"ID":"db088276-f18e-4fd4-ab03-8fb01725451e","Type":"ContainerDied","Data":"0ab9810899e62b9eb4738ce2a7f95cfc0d05d4746613910923e0c6800868d303"}
Jul 17 00:23:00.585176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.585148    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc" event={"ID":"db088276-f18e-4fd4-ab03-8fb01725451e","Type":"ContainerStarted","Data":"487713fa83ad269a0dd0770f507340cd98d986854ee4f02755f8acf4c8ba3744"}
Jul 17 00:23:00.621766 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.621739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.621927 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.621774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.621927 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.621840    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjdj\" (UniqueName: \"kubernetes.io/projected/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-kube-api-access-6xjdj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.622192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.622172    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.622269 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.622224    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.629243 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.629219    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjdj\" (UniqueName: \"kubernetes.io/projected/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-kube-api-access-6xjdj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.716267 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.716237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:00.789608 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.789576    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"]
Jul 17 00:23:00.794210 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.794190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:00.801704 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.801663    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"]
Jul 17 00:23:00.846514 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.846491    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"]
Jul 17 00:23:00.848412 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:00.848382    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b84bdb2_14fe_4da0_b015_f5c40e62a2e8.slice/crio-10215e2c24f994ab9ab4a659b89735f1ab469a8e2f0f94a82b1f38b37de05df7 WatchSource:0}: Error finding container 10215e2c24f994ab9ab4a659b89735f1ab469a8e2f0f94a82b1f38b37de05df7: Status 404 returned error can't find the container with id 10215e2c24f994ab9ab4a659b89735f1ab469a8e2f0f94a82b1f38b37de05df7
Jul 17 00:23:00.924979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.924952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9l6\" (UniqueName: \"kubernetes.io/projected/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-kube-api-access-6j9l6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:00.925119 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.924992    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:00.925119 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:00.925031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.025824 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.025777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9l6\" (UniqueName: \"kubernetes.io/projected/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-kube-api-access-6j9l6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.025927 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.025837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.025927 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.025907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.026299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.026273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.026388 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.026302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.033603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.033584    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9l6\" (UniqueName: \"kubernetes.io/projected/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-kube-api-access-6j9l6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.108323 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.108245    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:01.240209 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.240088    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"]
Jul 17 00:23:01.242324 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:01.242295    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aece1f6_a8c2_4eaa_948b_4f1e0bd20a41.slice/crio-c654808db7ab432d97fe6cebf5f02303c2694f78bf9a28fd6c1331c3df90f26f WatchSource:0}: Error finding container c654808db7ab432d97fe6cebf5f02303c2694f78bf9a28fd6c1331c3df90f26f: Status 404 returned error can't find the container with id c654808db7ab432d97fe6cebf5f02303c2694f78bf9a28fd6c1331c3df90f26f
Jul 17 00:23:01.591680 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.591647    2573 generic.go:356] "Generic (PLEG): container finished" podID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerID="3199f16f1a75382b1d73d3e69f2e450bfd7ccd4274b7fca40d3487c759d025d9" exitCode=0
Jul 17 00:23:01.592115 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.591727    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk" event={"ID":"84c1c8fb-4cef-450e-aa29-69685b2384e3","Type":"ContainerDied","Data":"3199f16f1a75382b1d73d3e69f2e450bfd7ccd4274b7fca40d3487c759d025d9"}
Jul 17 00:23:01.593220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.593202    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerID="3d8cd174f7cf011458b7d18b7782ffb7808c5859291339b7d677545677779bfc" exitCode=0
Jul 17 00:23:01.593316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.593289    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq" event={"ID":"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8","Type":"ContainerDied","Data":"3d8cd174f7cf011458b7d18b7782ffb7808c5859291339b7d677545677779bfc"}
Jul 17 00:23:01.593393 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.593378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq" event={"ID":"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8","Type":"ContainerStarted","Data":"10215e2c24f994ab9ab4a659b89735f1ab469a8e2f0f94a82b1f38b37de05df7"}
Jul 17 00:23:01.594766 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.594743    2573 generic.go:356] "Generic (PLEG): container finished" podID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerID="e2ce0b2cabdd6a0cefe83f935d6cd410a60418d4659ac23418cf49b4710e2081" exitCode=0
Jul 17 00:23:01.594852 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.594822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp" event={"ID":"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41","Type":"ContainerDied","Data":"e2ce0b2cabdd6a0cefe83f935d6cd410a60418d4659ac23418cf49b4710e2081"}
Jul 17 00:23:01.594897 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:01.594864    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp" event={"ID":"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41","Type":"ContainerStarted","Data":"c654808db7ab432d97fe6cebf5f02303c2694f78bf9a28fd6c1331c3df90f26f"}
Jul 17 00:23:02.603805 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:02.603766    2573 generic.go:356] "Generic (PLEG): container finished" podID="db088276-f18e-4fd4-ab03-8fb01725451e" containerID="d94759ff3a83ed555e0b567839b7625d0fef10c40f3169bf203414e018adadd5" exitCode=0
Jul 17 00:23:02.604159 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:02.603823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc" event={"ID":"db088276-f18e-4fd4-ab03-8fb01725451e","Type":"ContainerDied","Data":"d94759ff3a83ed555e0b567839b7625d0fef10c40f3169bf203414e018adadd5"}
Jul 17 00:23:02.606114 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:02.606090    2573 generic.go:356] "Generic (PLEG): container finished" podID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerID="280fb8643343437436a0f341788675cb26236f030de1ba3e41616ecea582276c" exitCode=0
Jul 17 00:23:02.606216 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:02.606131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk" event={"ID":"84c1c8fb-4cef-450e-aa29-69685b2384e3","Type":"ContainerDied","Data":"280fb8643343437436a0f341788675cb26236f030de1ba3e41616ecea582276c"}
Jul 17 00:23:03.613132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.613096    2573 generic.go:356] "Generic (PLEG): container finished" podID="db088276-f18e-4fd4-ab03-8fb01725451e" containerID="3cd241cf9d8ae4c4b8cfa0888f0db470c941c0defb17f7e91e607db1a7b36f55" exitCode=0
Jul 17 00:23:03.613541 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.613183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc" event={"ID":"db088276-f18e-4fd4-ab03-8fb01725451e","Type":"ContainerDied","Data":"3cd241cf9d8ae4c4b8cfa0888f0db470c941c0defb17f7e91e607db1a7b36f55"}
Jul 17 00:23:03.614807 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.614774    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerID="c13785cb32e65f739c105d0ba3619808190b303a497c4b5e5f1c96fbde70a2e8" exitCode=0
Jul 17 00:23:03.614913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.614835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq" event={"ID":"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8","Type":"ContainerDied","Data":"c13785cb32e65f739c105d0ba3619808190b303a497c4b5e5f1c96fbde70a2e8"}
Jul 17 00:23:03.616760 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.616737    2573 generic.go:356] "Generic (PLEG): container finished" podID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerID="615d1d6f0262b060bebf1a2dc0ba9e4f0ef7c42a173db0ef6bf2d1198ca59a25" exitCode=0
Jul 17 00:23:03.616885 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.616820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp" event={"ID":"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41","Type":"ContainerDied","Data":"615d1d6f0262b060bebf1a2dc0ba9e4f0ef7c42a173db0ef6bf2d1198ca59a25"}
Jul 17 00:23:03.760504 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.760475    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:03.849896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.849871    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpgj6\" (UniqueName: \"kubernetes.io/projected/84c1c8fb-4cef-450e-aa29-69685b2384e3-kube-api-access-jpgj6\") pod \"84c1c8fb-4cef-450e-aa29-69685b2384e3\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") "
Jul 17 00:23:03.849984 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.849916    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-bundle\") pod \"84c1c8fb-4cef-450e-aa29-69685b2384e3\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") "
Jul 17 00:23:03.850068 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.850052    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-util\") pod \"84c1c8fb-4cef-450e-aa29-69685b2384e3\" (UID: \"84c1c8fb-4cef-450e-aa29-69685b2384e3\") "
Jul 17 00:23:03.850555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.850532    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-bundle" (OuterVolumeSpecName: "bundle") pod "84c1c8fb-4cef-450e-aa29-69685b2384e3" (UID: "84c1c8fb-4cef-450e-aa29-69685b2384e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:03.851856 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.851836    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c1c8fb-4cef-450e-aa29-69685b2384e3-kube-api-access-jpgj6" (OuterVolumeSpecName: "kube-api-access-jpgj6") pod "84c1c8fb-4cef-450e-aa29-69685b2384e3" (UID: "84c1c8fb-4cef-450e-aa29-69685b2384e3"). InnerVolumeSpecName "kube-api-access-jpgj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:23:03.855640 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.855621    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-util" (OuterVolumeSpecName: "util") pod "84c1c8fb-4cef-450e-aa29-69685b2384e3" (UID: "84c1c8fb-4cef-450e-aa29-69685b2384e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:03.950696 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.950641    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpgj6\" (UniqueName: \"kubernetes.io/projected/84c1c8fb-4cef-450e-aa29-69685b2384e3-kube-api-access-jpgj6\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:03.950696 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.950663    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:03.950696 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:03.950672    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c1c8fb-4cef-450e-aa29-69685b2384e3-util\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:04.623880 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.623851    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk"
Jul 17 00:23:04.623880 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.623856    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk" event={"ID":"84c1c8fb-4cef-450e-aa29-69685b2384e3","Type":"ContainerDied","Data":"821816f09b6fd11ee3568210edf6937ad5ba1b0584d0d74c77c566a6aa5b3c54"}
Jul 17 00:23:04.624315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.623908    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="821816f09b6fd11ee3568210edf6937ad5ba1b0584d0d74c77c566a6aa5b3c54"
Jul 17 00:23:04.626080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.626049    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerID="0ec8aa5fa5b0426919cacfbcced7309da86a783fdf633951b1f78afb7e1e5abd" exitCode=0
Jul 17 00:23:04.626212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.626139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq" event={"ID":"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8","Type":"ContainerDied","Data":"0ec8aa5fa5b0426919cacfbcced7309da86a783fdf633951b1f78afb7e1e5abd"}
Jul 17 00:23:04.628185 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.628159    2573 generic.go:356] "Generic (PLEG): container finished" podID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerID="1ae71a60b2abd6e9cff5f89ce5de08ed1bbd96a9834f2619ce8d2eb9f2b2144d" exitCode=0
Jul 17 00:23:04.628304 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.628224    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp" event={"ID":"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41","Type":"ContainerDied","Data":"1ae71a60b2abd6e9cff5f89ce5de08ed1bbd96a9834f2619ce8d2eb9f2b2144d"}
Jul 17 00:23:04.758854 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.758831    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:23:04.856190 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.856164    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-util\") pod \"db088276-f18e-4fd4-ab03-8fb01725451e\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") "
Jul 17 00:23:04.856336 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.856198    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrp7d\" (UniqueName: \"kubernetes.io/projected/db088276-f18e-4fd4-ab03-8fb01725451e-kube-api-access-zrp7d\") pod \"db088276-f18e-4fd4-ab03-8fb01725451e\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") "
Jul 17 00:23:04.856336 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.856227    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-bundle\") pod \"db088276-f18e-4fd4-ab03-8fb01725451e\" (UID: \"db088276-f18e-4fd4-ab03-8fb01725451e\") "
Jul 17 00:23:04.856643 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.856621    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-bundle" (OuterVolumeSpecName: "bundle") pod "db088276-f18e-4fd4-ab03-8fb01725451e" (UID: "db088276-f18e-4fd4-ab03-8fb01725451e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:04.858170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.858146    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db088276-f18e-4fd4-ab03-8fb01725451e-kube-api-access-zrp7d" (OuterVolumeSpecName: "kube-api-access-zrp7d") pod "db088276-f18e-4fd4-ab03-8fb01725451e" (UID: "db088276-f18e-4fd4-ab03-8fb01725451e"). InnerVolumeSpecName "kube-api-access-zrp7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:23:04.861651 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.861615    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-util" (OuterVolumeSpecName: "util") pod "db088276-f18e-4fd4-ab03-8fb01725451e" (UID: "db088276-f18e-4fd4-ab03-8fb01725451e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:04.957710 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.957644    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrp7d\" (UniqueName: \"kubernetes.io/projected/db088276-f18e-4fd4-ab03-8fb01725451e-kube-api-access-zrp7d\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:04.957710 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.957670    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:04.957710 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:04.957682    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db088276-f18e-4fd4-ab03-8fb01725451e-util\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:05.634715 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.634688    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc"
Jul 17 00:23:05.635103 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.634713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc" event={"ID":"db088276-f18e-4fd4-ab03-8fb01725451e","Type":"ContainerDied","Data":"487713fa83ad269a0dd0770f507340cd98d986854ee4f02755f8acf4c8ba3744"}
Jul 17 00:23:05.635103 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.634767    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="487713fa83ad269a0dd0770f507340cd98d986854ee4f02755f8acf4c8ba3744"
Jul 17 00:23:05.788010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.787988    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:05.789952 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.789931    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:05.865404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.865379    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-util\") pod \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") "
Jul 17 00:23:05.865548 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.865421    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjdj\" (UniqueName: \"kubernetes.io/projected/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-kube-api-access-6xjdj\") pod \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") "
Jul 17 00:23:05.865548 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.865476    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-bundle\") pod \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") "
Jul 17 00:23:05.865548 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.865491    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-util\") pod \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") "
Jul 17 00:23:05.865548 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.865508    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j9l6\" (UniqueName: \"kubernetes.io/projected/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-kube-api-access-6j9l6\") pod \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\" (UID: \"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41\") "
Jul 17 00:23:05.865753 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.865576    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-bundle\") pod \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\" (UID: \"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8\") "
Jul 17 00:23:05.866138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.866078    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-bundle" (OuterVolumeSpecName: "bundle") pod "2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" (UID: "2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:05.866312 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.866287    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-bundle" (OuterVolumeSpecName: "bundle") pod "1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" (UID: "1b84bdb2-14fe-4da0-b015-f5c40e62a2e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:05.867675 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.867646    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-kube-api-access-6j9l6" (OuterVolumeSpecName: "kube-api-access-6j9l6") pod "2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" (UID: "2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41"). InnerVolumeSpecName "kube-api-access-6j9l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:23:05.867908 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.867885    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-kube-api-access-6xjdj" (OuterVolumeSpecName: "kube-api-access-6xjdj") pod "1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" (UID: "1b84bdb2-14fe-4da0-b015-f5c40e62a2e8"). InnerVolumeSpecName "kube-api-access-6xjdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:23:05.870979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.870956    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-util" (OuterVolumeSpecName: "util") pod "1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" (UID: "1b84bdb2-14fe-4da0-b015-f5c40e62a2e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:05.871400 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.871383    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-util" (OuterVolumeSpecName: "util") pod "2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" (UID: "2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:05.969469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.967529    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-util\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:05.969469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.967563    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjdj\" (UniqueName: \"kubernetes.io/projected/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-kube-api-access-6xjdj\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:05.969469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.967578    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:05.969469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.967591    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-util\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:05.969469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.967604    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9l6\" (UniqueName: \"kubernetes.io/projected/2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41-kube-api-access-6j9l6\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:05.969469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:05.967620    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b84bdb2-14fe-4da0-b015-f5c40e62a2e8-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:06.641665 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:06.641640    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq"
Jul 17 00:23:06.641665 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:06.641650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq" event={"ID":"1b84bdb2-14fe-4da0-b015-f5c40e62a2e8","Type":"ContainerDied","Data":"10215e2c24f994ab9ab4a659b89735f1ab469a8e2f0f94a82b1f38b37de05df7"}
Jul 17 00:23:06.642150 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:06.641701    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10215e2c24f994ab9ab4a659b89735f1ab469a8e2f0f94a82b1f38b37de05df7"
Jul 17 00:23:06.643937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:06.643841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp" event={"ID":"2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41","Type":"ContainerDied","Data":"c654808db7ab432d97fe6cebf5f02303c2694f78bf9a28fd6c1331c3df90f26f"}
Jul 17 00:23:06.643937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:06.643889    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c654808db7ab432d97fe6cebf5f02303c2694f78bf9a28fd6c1331c3df90f26f"
Jul 17 00:23:06.643937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:06.643928    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp"
Jul 17 00:23:16.416588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416549    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"]
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416888    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerName="util"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416904    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerName="util"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416915    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db088276-f18e-4fd4-ab03-8fb01725451e" containerName="pull"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416920    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db088276-f18e-4fd4-ab03-8fb01725451e" containerName="pull"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416933    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerName="util"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416940    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerName="util"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416948    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db088276-f18e-4fd4-ab03-8fb01725451e" containerName="util"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416953    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db088276-f18e-4fd4-ab03-8fb01725451e" containerName="util"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416959    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db088276-f18e-4fd4-ab03-8fb01725451e" containerName="extract"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416965    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db088276-f18e-4fd4-ab03-8fb01725451e" containerName="extract"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416975    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerName="pull"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416984    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerName="pull"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416994    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerName="extract"
Jul 17 00:23:16.416991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.416999    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417006    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerName="pull"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417011    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerName="pull"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417016    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417021    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417028    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417035    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417043    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerName="util"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417048    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerName="util"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417055    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerName="pull"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417063    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerName="pull"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417111    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c1c8fb-4cef-450e-aa29-69685b2384e3" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417119    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417125    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="db088276-f18e-4fd4-ab03-8fb01725451e" containerName="extract"
Jul 17 00:23:16.417378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.417131    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b84bdb2-14fe-4da0-b015-f5c40e62a2e8" containerName="extract"
Jul 17 00:23:16.420120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.420097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"
Jul 17 00:23:16.425814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.425771    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dfsjs\""
Jul 17 00:23:16.430320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.430288    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"]
Jul 17 00:23:16.546133 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.546104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkw29\" (UniqueName: \"kubernetes.io/projected/f7b46a10-a42d-40b5-8ad6-986d6740be80-kube-api-access-kkw29\") pod \"authorino-operator-75cc8d5cb7-gr8xz\" (UID: \"f7b46a10-a42d-40b5-8ad6-986d6740be80\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"
Jul 17 00:23:16.646513 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.646475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkw29\" (UniqueName: \"kubernetes.io/projected/f7b46a10-a42d-40b5-8ad6-986d6740be80-kube-api-access-kkw29\") pod \"authorino-operator-75cc8d5cb7-gr8xz\" (UID: \"f7b46a10-a42d-40b5-8ad6-986d6740be80\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"
Jul 17 00:23:16.657407 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.657382    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkw29\" (UniqueName: \"kubernetes.io/projected/f7b46a10-a42d-40b5-8ad6-986d6740be80-kube-api-access-kkw29\") pod \"authorino-operator-75cc8d5cb7-gr8xz\" (UID: \"f7b46a10-a42d-40b5-8ad6-986d6740be80\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"
Jul 17 00:23:16.731247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.731180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"
Jul 17 00:23:16.862575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:16.860573    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"]
Jul 17 00:23:16.863572 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:16.863545    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b46a10_a42d_40b5_8ad6_986d6740be80.slice/crio-7de32432905b43d74acd8051bcf547d12aba33c019625c67af6c4daa777103ea WatchSource:0}: Error finding container 7de32432905b43d74acd8051bcf547d12aba33c019625c67af6c4daa777103ea: Status 404 returned error can't find the container with id 7de32432905b43d74acd8051bcf547d12aba33c019625c67af6c4daa777103ea
Jul 17 00:23:17.623677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.623409    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79bfbfc896-hk6zb"]
Jul 17 00:23:17.627218 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.627195    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.648514 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.648482    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bfbfc896-hk6zb"]
Jul 17 00:23:17.698605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.698561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz" event={"ID":"f7b46a10-a42d-40b5-8ad6-986d6740be80","Type":"ContainerStarted","Data":"7de32432905b43d74acd8051bcf547d12aba33c019625c67af6c4daa777103ea"}
Jul 17 00:23:17.756167 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.756130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfrz\" (UniqueName: \"kubernetes.io/projected/bf073599-1ef5-4774-88fd-c13a7d50266d-kube-api-access-czfrz\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.756303 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.756174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf073599-1ef5-4774-88fd-c13a7d50266d-console-oauth-config\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.756303 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.756195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-trusted-ca-bundle\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.756303 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.756223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-service-ca\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.756303 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.756256    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-oauth-serving-cert\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.756506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.756325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf073599-1ef5-4774-88fd-c13a7d50266d-console-serving-cert\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.756506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.756389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-console-config\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.857603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.857567    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-console-config\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.857774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.857618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfrz\" (UniqueName: \"kubernetes.io/projected/bf073599-1ef5-4774-88fd-c13a7d50266d-kube-api-access-czfrz\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.857774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.857646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf073599-1ef5-4774-88fd-c13a7d50266d-console-oauth-config\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.857774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.857669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-trusted-ca-bundle\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.857774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.857692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-service-ca\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.857774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.857720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-oauth-serving-cert\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.858068 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.857784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf073599-1ef5-4774-88fd-c13a7d50266d-console-serving-cert\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.858435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.858402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-console-config\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.858590 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.858563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-service-ca\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.858701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.858603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-oauth-serving-cert\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.858878 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.858857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf073599-1ef5-4774-88fd-c13a7d50266d-trusted-ca-bundle\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.860630 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.860603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf073599-1ef5-4774-88fd-c13a7d50266d-console-serving-cert\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.860716 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.860664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf073599-1ef5-4774-88fd-c13a7d50266d-console-oauth-config\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.876781 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.876718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfrz\" (UniqueName: \"kubernetes.io/projected/bf073599-1ef5-4774-88fd-c13a7d50266d-kube-api-access-czfrz\") pod \"console-79bfbfc896-hk6zb\" (UID: \"bf073599-1ef5-4774-88fd-c13a7d50266d\") " pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:17.940755 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:17.940721    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:18.099966 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:18.099936    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bfbfc896-hk6zb"]
Jul 17 00:23:18.918007 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:18.917973    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf073599_1ef5_4774_88fd_c13a7d50266d.slice/crio-494b7e4254843b2fb2f94a836e975e0999594ee90ac91e0ffa4e2f1d66912010 WatchSource:0}: Error finding container 494b7e4254843b2fb2f94a836e975e0999594ee90ac91e0ffa4e2f1d66912010: Status 404 returned error can't find the container with id 494b7e4254843b2fb2f94a836e975e0999594ee90ac91e0ffa4e2f1d66912010
Jul 17 00:23:19.169686 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.169600    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"]
Jul 17 00:23:19.172905 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.172883    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"
Jul 17 00:23:19.175296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.175277    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-khkmm\""
Jul 17 00:23:19.175459 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.175441    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 17 00:23:19.182582 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.182558    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"]
Jul 17 00:23:19.271530 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.271494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5pg\" (UniqueName: \"kubernetes.io/projected/750eefa1-5519-4cb9-bebc-58e9d2a55d48-kube-api-access-sb5pg\") pod \"dns-operator-controller-manager-864dd78c4c-rjmbk\" (UID: \"750eefa1-5519-4cb9-bebc-58e9d2a55d48\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"
Jul 17 00:23:19.372306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.372280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5pg\" (UniqueName: \"kubernetes.io/projected/750eefa1-5519-4cb9-bebc-58e9d2a55d48-kube-api-access-sb5pg\") pod \"dns-operator-controller-manager-864dd78c4c-rjmbk\" (UID: \"750eefa1-5519-4cb9-bebc-58e9d2a55d48\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"
Jul 17 00:23:19.383044 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.383020    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5pg\" (UniqueName: \"kubernetes.io/projected/750eefa1-5519-4cb9-bebc-58e9d2a55d48-kube-api-access-sb5pg\") pod \"dns-operator-controller-manager-864dd78c4c-rjmbk\" (UID: \"750eefa1-5519-4cb9-bebc-58e9d2a55d48\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"
Jul 17 00:23:19.484986 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.484902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"
Jul 17 00:23:19.619362 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.619336    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"]
Jul 17 00:23:19.621469 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:19.621434    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750eefa1_5519_4cb9_bebc_58e9d2a55d48.slice/crio-3dc54057b088d57f0dc5b35b96682c76165e395943de338a5303317c68311c62 WatchSource:0}: Error finding container 3dc54057b088d57f0dc5b35b96682c76165e395943de338a5303317c68311c62: Status 404 returned error can't find the container with id 3dc54057b088d57f0dc5b35b96682c76165e395943de338a5303317c68311c62
Jul 17 00:23:19.713007 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.712975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz" event={"ID":"f7b46a10-a42d-40b5-8ad6-986d6740be80","Type":"ContainerStarted","Data":"a41fecc24b7b323ff3fbeb31a4b4e3d0a5cfce9d6009a0a10e87d3b892c84325"}
Jul 17 00:23:19.713171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.713088    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"
Jul 17 00:23:19.714241 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.714216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk" event={"ID":"750eefa1-5519-4cb9-bebc-58e9d2a55d48","Type":"ContainerStarted","Data":"3dc54057b088d57f0dc5b35b96682c76165e395943de338a5303317c68311c62"}
Jul 17 00:23:19.715677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.715651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bfbfc896-hk6zb" event={"ID":"bf073599-1ef5-4774-88fd-c13a7d50266d","Type":"ContainerStarted","Data":"b2b9a2204da55d1f03ceee99e02eb7af9d5d20ded276cdbc0275dd49bfcbf948"}
Jul 17 00:23:19.715766 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.715689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bfbfc896-hk6zb" event={"ID":"bf073599-1ef5-4774-88fd-c13a7d50266d","Type":"ContainerStarted","Data":"494b7e4254843b2fb2f94a836e975e0999594ee90ac91e0ffa4e2f1d66912010"}
Jul 17 00:23:19.729406 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.729364    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz" podStartSLOduration=1.625253024 podStartE2EDuration="3.729350596s" totalImagesPullingTime="2.104097572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:19.727637381 +0000 UTC m=+394.303789957" watchObservedRunningTime="2026-07-17 00:23:19.729350596 +0000 UTC m=+394.305503169"
Jul 17 00:23:19.748957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:19.748874    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79bfbfc896-hk6zb" podStartSLOduration=2.7488588910000002 podStartE2EDuration="2.748858891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:19.747557153 +0000 UTC m=+394.323709726" watchObservedRunningTime="2026-07-17 00:23:19.748858891 +0000 UTC m=+394.325011464"
Jul 17 00:23:22.733530 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:22.733493    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk" event={"ID":"750eefa1-5519-4cb9-bebc-58e9d2a55d48","Type":"ContainerStarted","Data":"a9f7a369516f041619ca663383508656c49579ea813ae4aa09245b037e69b78e"}
Jul 17 00:23:22.733962 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:22.733627    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"
Jul 17 00:23:22.754654 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:22.754609    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk" podStartSLOduration=1.561338417 podStartE2EDuration="3.754592859s" totalImagesPullingTime="2.193254442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:22.752551511 +0000 UTC m=+397.328704086" watchObservedRunningTime="2026-07-17 00:23:22.754592859 +0000 UTC m=+397.330745434"
Jul 17 00:23:23.398202 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.398168    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"]
Jul 17 00:23:23.401494 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.401469    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.404421 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.404405    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jg6jl\""
Jul 17 00:23:23.413994 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.413974    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"]
Jul 17 00:23:23.509671 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.509629    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgkx\" (UniqueName: \"kubernetes.io/projected/e0801a73-41e7-4d9c-a7f0-5de157467d96-kube-api-access-tzgkx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.509883 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.509769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0801a73-41e7-4d9c-a7f0-5de157467d96-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.610768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.610738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0801a73-41e7-4d9c-a7f0-5de157467d96-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.610949 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.610812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgkx\" (UniqueName: \"kubernetes.io/projected/e0801a73-41e7-4d9c-a7f0-5de157467d96-kube-api-access-tzgkx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.611211 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.611190    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0801a73-41e7-4d9c-a7f0-5de157467d96-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.619364 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.619334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgkx\" (UniqueName: \"kubernetes.io/projected/e0801a73-41e7-4d9c-a7f0-5de157467d96-kube-api-access-tzgkx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.711379 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.711293    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:23.841536 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:23.841501    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"]
Jul 17 00:23:23.843289 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:23.843257    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0801a73_41e7_4d9c_a7f0_5de157467d96.slice/crio-54ec3a39c010f6a01672292dc60d326a20ceb2af26e2e653c7c39ce9dd677ee8 WatchSource:0}: Error finding container 54ec3a39c010f6a01672292dc60d326a20ceb2af26e2e653c7c39ce9dd677ee8: Status 404 returned error can't find the container with id 54ec3a39c010f6a01672292dc60d326a20ceb2af26e2e653c7c39ce9dd677ee8
Jul 17 00:23:24.745117 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:24.745077    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj" event={"ID":"e0801a73-41e7-4d9c-a7f0-5de157467d96","Type":"ContainerStarted","Data":"54ec3a39c010f6a01672292dc60d326a20ceb2af26e2e653c7c39ce9dd677ee8"}
Jul 17 00:23:27.941351 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:27.941308    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:27.941720 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:27.941383    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:27.945727 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:27.945704    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:28.768285 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:28.768249    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79bfbfc896-hk6zb"
Jul 17 00:23:28.815358 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:28.815331    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647467766c-jq7gg"]
Jul 17 00:23:30.722940 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:30.722907    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gr8xz"
Jul 17 00:23:31.781837 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:31.781774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj" event={"ID":"e0801a73-41e7-4d9c-a7f0-5de157467d96","Type":"ContainerStarted","Data":"3324fd410761caf92c42b2aae95073c59048628c1567af03d98673069f00557a"}
Jul 17 00:23:31.782219 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:31.781883    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:31.800332 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:31.800281    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj" podStartSLOduration=1.887786002 podStartE2EDuration="8.800263879s" totalImagesPullingTime="6.912477877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:31.798783899 +0000 UTC m=+406.374936473" watchObservedRunningTime="2026-07-17 00:23:31.800263879 +0000 UTC m=+406.376416455"
Jul 17 00:23:33.574475 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.574442    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"]
Jul 17 00:23:33.578020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.577998    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.580020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.579994    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 17 00:23:33.580134 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.580046    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 17 00:23:33.580315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.580296    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8kgg7\""
Jul 17 00:23:33.593416 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.593394    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"]
Jul 17 00:23:33.701523 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.701498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c02072b4-cf4f-4566-8fc0-e5b552507e54-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.701643 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.701548    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttkv2\" (UniqueName: \"kubernetes.io/projected/c02072b4-cf4f-4566-8fc0-e5b552507e54-kube-api-access-ttkv2\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.701643 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.701633    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c02072b4-cf4f-4566-8fc0-e5b552507e54-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.740894 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.740876    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rjmbk"
Jul 17 00:23:33.802048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.802024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c02072b4-cf4f-4566-8fc0-e5b552507e54-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.802171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.802081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c02072b4-cf4f-4566-8fc0-e5b552507e54-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.802171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.802132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttkv2\" (UniqueName: \"kubernetes.io/projected/c02072b4-cf4f-4566-8fc0-e5b552507e54-kube-api-access-ttkv2\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.802342 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:33.802205    2573 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 17 00:23:33.802342 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:33.802285    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02072b4-cf4f-4566-8fc0-e5b552507e54-plugin-serving-cert podName:c02072b4-cf4f-4566-8fc0-e5b552507e54 nodeName:}" failed. No retries permitted until 2026-07-17 00:23:34.302261787 +0000 UTC m=+408.878414341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c02072b4-cf4f-4566-8fc0-e5b552507e54-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-kszxw" (UID: "c02072b4-cf4f-4566-8fc0-e5b552507e54") : secret "plugin-serving-cert" not found
Jul 17 00:23:33.802713 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.802696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c02072b4-cf4f-4566-8fc0-e5b552507e54-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:33.810199 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:33.810175    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttkv2\" (UniqueName: \"kubernetes.io/projected/c02072b4-cf4f-4566-8fc0-e5b552507e54-kube-api-access-ttkv2\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:34.304874 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:34.304841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c02072b4-cf4f-4566-8fc0-e5b552507e54-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:34.307230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:34.307211    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c02072b4-cf4f-4566-8fc0-e5b552507e54-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszxw\" (UID: \"c02072b4-cf4f-4566-8fc0-e5b552507e54\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:34.486874 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:34.486835    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"
Jul 17 00:23:34.613905 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:34.613881    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw"]
Jul 17 00:23:34.615668 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:34.615643    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc02072b4_cf4f_4566_8fc0_e5b552507e54.slice/crio-6fa17e84bebd0103d9662f70da1d44cd7b5e8a7b7c26046ba06fc6f2270d2f3f WatchSource:0}: Error finding container 6fa17e84bebd0103d9662f70da1d44cd7b5e8a7b7c26046ba06fc6f2270d2f3f: Status 404 returned error can't find the container with id 6fa17e84bebd0103d9662f70da1d44cd7b5e8a7b7c26046ba06fc6f2270d2f3f
Jul 17 00:23:34.797404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:34.797370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw" event={"ID":"c02072b4-cf4f-4566-8fc0-e5b552507e54","Type":"ContainerStarted","Data":"6fa17e84bebd0103d9662f70da1d44cd7b5e8a7b7c26046ba06fc6f2270d2f3f"}
Jul 17 00:23:42.790021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:42.789986    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:43.750341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.750308    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"]
Jul 17 00:23:43.770421 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.770370    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"]
Jul 17 00:23:43.770576 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.770484    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:43.891442 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.891410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfs5p\" (UniqueName: \"kubernetes.io/projected/0dd7f729-5102-492b-b5bb-e31e56e102b5-kube-api-access-zfs5p\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:43.891861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.891541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd7f729-5102-492b-b5bb-e31e56e102b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:43.992648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.992616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd7f729-5102-492b-b5bb-e31e56e102b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:43.992836 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.992729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfs5p\" (UniqueName: \"kubernetes.io/projected/0dd7f729-5102-492b-b5bb-e31e56e102b5-kube-api-access-zfs5p\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:43.993083 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:43.993054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd7f729-5102-492b-b5bb-e31e56e102b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:44.016606 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.016585    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfs5p\" (UniqueName: \"kubernetes.io/projected/0dd7f729-5102-492b-b5bb-e31e56e102b5-kube-api-access-zfs5p\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:44.082629 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.082592    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:44.390978 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.390902    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"]
Jul 17 00:23:44.395094 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.395066    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"]
Jul 17 00:23:44.402762 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.402736    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"]
Jul 17 00:23:44.402951 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.402923    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" containerName="manager" containerID="cri-o://3324fd410761caf92c42b2aae95073c59048628c1567af03d98673069f00557a" gracePeriod=2
Jul 17 00:23:44.414221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.414111    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"]
Jul 17 00:23:44.417339 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.416917    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"]
Jul 17 00:23:44.417448 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.417396    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" containerName="manager"
Jul 17 00:23:44.417448 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.417416    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" containerName="manager"
Jul 17 00:23:44.417563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.417508    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" containerName="manager"
Jul 17 00:23:44.422061 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:44.422015    2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e0801a73-41e7-4d9c-a7f0-5de157467d96\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-tvffj\": pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:44.424074 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.424051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.429212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.429185    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"]
Jul 17 00:23:44.434215 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.434193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:44.445847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.445823    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"]
Jul 17 00:23:44.455171 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:44.455112    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:44.456855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.456831    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"]
Jul 17 00:23:44.474917 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:44.474876    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:44.496103 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.496076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rt8xl\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:44.496222 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.496115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jxd\" (UniqueName: \"kubernetes.io/projected/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-kube-api-access-h6jxd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rt8xl\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:44.496222 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.496190    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5w9t\" (UniqueName: \"kubernetes.io/projected/01d6062e-fbd8-47a6-af89-f25326194254-kube-api-access-x5w9t\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ks7mv\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.496336 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.496277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d6062e-fbd8-47a6-af89-f25326194254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ks7mv\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.597450 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.597418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rt8xl\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:44.597697 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.597460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jxd\" (UniqueName: \"kubernetes.io/projected/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-kube-api-access-h6jxd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rt8xl\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:44.597697 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.597513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5w9t\" (UniqueName: \"kubernetes.io/projected/01d6062e-fbd8-47a6-af89-f25326194254-kube-api-access-x5w9t\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ks7mv\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.597697 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.597575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d6062e-fbd8-47a6-af89-f25326194254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ks7mv\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.597923 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.597862    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rt8xl\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:44.597987 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.597964    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d6062e-fbd8-47a6-af89-f25326194254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ks7mv\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.606449 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.606423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5w9t\" (UniqueName: \"kubernetes.io/projected/01d6062e-fbd8-47a6-af89-f25326194254-kube-api-access-x5w9t\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ks7mv\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.606627 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.606604    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jxd\" (UniqueName: \"kubernetes.io/projected/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-kube-api-access-h6jxd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rt8xl\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:44.777941 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.777902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:44.785542 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:44.785497    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:45.969744 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:45.969697    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:53.835962 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:53.835911    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-647467766c-jq7gg" podUID="1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" containerName="console" containerID="cri-o://2707bce0e149c638369fd572071ed96d48cab979d4b954f2183096ba5f449459" gracePeriod=15
Jul 17 00:23:55.935488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:55.935288    2573 generic.go:356] "Generic (PLEG): container finished" podID="e0801a73-41e7-4d9c-a7f0-5de157467d96" containerID="3324fd410761caf92c42b2aae95073c59048628c1567af03d98673069f00557a" exitCode=0
Jul 17 00:23:55.938090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:55.937950    2573 generic.go:356] "Generic (PLEG): container finished" podID="1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" containerID="2707bce0e149c638369fd572071ed96d48cab979d4b954f2183096ba5f449459" exitCode=0
Jul 17 00:23:55.938090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:55.938009    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647467766c-jq7gg" event={"ID":"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875","Type":"ContainerDied","Data":"2707bce0e149c638369fd572071ed96d48cab979d4b954f2183096ba5f449459"}
Jul 17 00:23:55.997537 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:55.997488    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:55.999413 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:55.999378    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:55.999781 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:55.999759    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:23:56.001319 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:56.001289    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:56.025461 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.025433    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"]
Jul 17 00:23:56.029807 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:23:56.029766    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d6062e_fbd8_47a6_af89_f25326194254.slice/crio-451b374a96c4cf2e4b78f475988703092c96c043f6397acbf267a3d03ab50468 WatchSource:0}: Error finding container 451b374a96c4cf2e4b78f475988703092c96c043f6397acbf267a3d03ab50468: Status 404 returned error can't find the container with id 451b374a96c4cf2e4b78f475988703092c96c043f6397acbf267a3d03ab50468
Jul 17 00:23:56.052306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.052275    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"]
Jul 17 00:23:56.105562 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105539    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zmrt\" (UniqueName: \"kubernetes.io/projected/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-kube-api-access-5zmrt\") pod \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") "
Jul 17 00:23:56.105669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105569    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-oauth-serving-cert\") pod \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") "
Jul 17 00:23:56.105669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105606    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-service-ca\") pod \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") "
Jul 17 00:23:56.105669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105664    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-config\") pod \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") "
Jul 17 00:23:56.105847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105696    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzgkx\" (UniqueName: \"kubernetes.io/projected/e0801a73-41e7-4d9c-a7f0-5de157467d96-kube-api-access-tzgkx\") pod \"e0801a73-41e7-4d9c-a7f0-5de157467d96\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") "
Jul 17 00:23:56.105847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105732    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-serving-cert\") pod \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") "
Jul 17 00:23:56.105847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105821    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-trusted-ca-bundle\") pod \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") "
Jul 17 00:23:56.106017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105852    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0801a73-41e7-4d9c-a7f0-5de157467d96-extensions-socket-volume\") pod \"e0801a73-41e7-4d9c-a7f0-5de157467d96\" (UID: \"e0801a73-41e7-4d9c-a7f0-5de157467d96\") "
Jul 17 00:23:56.106017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105881    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-oauth-config\") pod \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\" (UID: \"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875\") "
Jul 17 00:23:56.106017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.105999    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" (UID: "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:23:56.106170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.106026    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-service-ca" (OuterVolumeSpecName: "service-ca") pod "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" (UID: "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:23:56.106170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.106151    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-oauth-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.106170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.106167    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.106653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.106481    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" (UID: "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:23:56.106653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.106534    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0801a73-41e7-4d9c-a7f0-5de157467d96-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e0801a73-41e7-4d9c-a7f0-5de157467d96" (UID: "e0801a73-41e7-4d9c-a7f0-5de157467d96"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:56.106653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.106620    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-config" (OuterVolumeSpecName: "console-config") pod "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" (UID: "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:23:56.108441 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.108403    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0801a73-41e7-4d9c-a7f0-5de157467d96-kube-api-access-tzgkx" (OuterVolumeSpecName: "kube-api-access-tzgkx") pod "e0801a73-41e7-4d9c-a7f0-5de157467d96" (UID: "e0801a73-41e7-4d9c-a7f0-5de157467d96"). InnerVolumeSpecName "kube-api-access-tzgkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:23:56.108587 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.108569    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-kube-api-access-5zmrt" (OuterVolumeSpecName: "kube-api-access-5zmrt") pod "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" (UID: "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875"). InnerVolumeSpecName "kube-api-access-5zmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:23:56.108668 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.108653    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" (UID: "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:23:56.108716 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.108692    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" (UID: "1eea02e1-aeb2-48eb-89c6-b6d11fb3b875"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:23:56.207509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.207473    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-trusted-ca-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.207509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.207501    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0801a73-41e7-4d9c-a7f0-5de157467d96-extensions-socket-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.207509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.207511    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-oauth-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.207728 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.207519    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmrt\" (UniqueName: \"kubernetes.io/projected/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-kube-api-access-5zmrt\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.207728 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.207529    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.207728 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.207536    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgkx\" (UniqueName: \"kubernetes.io/projected/e0801a73-41e7-4d9c-a7f0-5de157467d96-kube-api-access-tzgkx\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.207728 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.207544    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875-console-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:56.946109 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.946076    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647467766c-jq7gg"
Jul 17 00:23:56.946109 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.946091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647467766c-jq7gg" event={"ID":"1eea02e1-aeb2-48eb-89c6-b6d11fb3b875","Type":"ContainerDied","Data":"d1bb4d7cdb9182889df2d9351333409a23085c3a1474edc24546d453d3ae2007"}
Jul 17 00:23:56.946639 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.946156    2573 scope.go:117] "RemoveContainer" containerID="2707bce0e149c638369fd572071ed96d48cab979d4b954f2183096ba5f449459"
Jul 17 00:23:56.947776 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.947750    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:56.949609 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.949581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl" event={"ID":"2d25088e-ff2a-470b-b536-9b11d4e9ef7f","Type":"ContainerStarted","Data":"ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669"}
Jul 17 00:23:56.949725 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.949633    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl" event={"ID":"2d25088e-ff2a-470b-b536-9b11d4e9ef7f","Type":"ContainerStarted","Data":"7a590a302f774fb68d2a396e20a1cffcf8b622c167988a279c3ff228aafa88a3"}
Jul 17 00:23:56.949725 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.949693    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:23:56.950712 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:56.950668    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:56.951512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.951489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw" event={"ID":"c02072b4-cf4f-4566-8fc0-e5b552507e54","Type":"ContainerStarted","Data":"3ae8c6215846e67dc5da4352171d4c3035c1c697c9baf6415882dffdef8ef8fb"}
Jul 17 00:23:56.952255 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:56.952227    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:56.953809 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.953762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv" event={"ID":"01d6062e-fbd8-47a6-af89-f25326194254","Type":"ContainerStarted","Data":"0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320"}
Jul 17 00:23:56.953911 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.953836    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv" event={"ID":"01d6062e-fbd8-47a6-af89-f25326194254","Type":"ContainerStarted","Data":"451b374a96c4cf2e4b78f475988703092c96c043f6397acbf267a3d03ab50468"}
Jul 17 00:23:56.953977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.953929    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:23:56.957469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.957439    2573 scope.go:117] "RemoveContainer" containerID="3324fd410761caf92c42b2aae95073c59048628c1567af03d98673069f00557a"
Jul 17 00:23:56.974348 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.974295    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl" podStartSLOduration=12.974277387 podStartE2EDuration="12.974277387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:56.971365054 +0000 UTC m=+431.547517631" watchObservedRunningTime="2026-07-17 00:23:56.974277387 +0000 UTC m=+431.550429966"
Jul 17 00:23:56.988504 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.988480    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647467766c-jq7gg"]
Jul 17 00:23:56.989744 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:56.989720    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tvffj\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvffj"
Jul 17 00:23:56.990348 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:56.990329    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-647467766c-jq7gg"]
Jul 17 00:23:57.006406 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:57.006368    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszxw" podStartSLOduration=2.720509895 podStartE2EDuration="24.00635574s" totalImagesPullingTime="21.285845845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:57.005224763 +0000 UTC m=+431.581377372" watchObservedRunningTime="2026-07-17 00:23:57.00635574 +0000 UTC m=+431.582508374"
Jul 17 00:23:57.025155 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:57.025117    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv" podStartSLOduration=13.025104796 podStartE2EDuration="13.025104796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:57.023887075 +0000 UTC m=+431.600039650" watchObservedRunningTime="2026-07-17 00:23:57.025104796 +0000 UTC m=+431.601257370"
Jul 17 00:23:57.967634 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:57.967600    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" path="/var/lib/kubelet/pods/1eea02e1-aeb2-48eb-89c6-b6d11fb3b875/volumes"
Jul 17 00:23:57.968139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:57.968118    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0801a73-41e7-4d9c-a7f0-5de157467d96" path="/var/lib/kubelet/pods/e0801a73-41e7-4d9c-a7f0-5de157467d96/volumes"
Jul 17 00:23:58.397486 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:58.397422    2573 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-9gxkp_kuadrant-system_0dd7f729-5102-492b-b5bb-e31e56e102b5_0(53128ba63d14247f204244cc4830cff44333a69aa3c4b2a0742fa435eb8217b8): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-9gxkp to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 17 00:23:58.397560 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:58.397515    2573 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-9gxkp_kuadrant-system_0dd7f729-5102-492b-b5bb-e31e56e102b5_0(53128ba63d14247f204244cc4830cff44333a69aa3c4b2a0742fa435eb8217b8): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-9gxkp to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:58.965854 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:58.965823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:58.967602 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:58.967572    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="0dd7f729-5102-492b-b5bb-e31e56e102b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:58.973348 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:58.973329    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:58.974972 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:58.974948    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="0dd7f729-5102-492b-b5bb-e31e56e102b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:59.031430 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.031401    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfs5p\" (UniqueName: \"kubernetes.io/projected/0dd7f729-5102-492b-b5bb-e31e56e102b5-kube-api-access-zfs5p\") pod \"0dd7f729-5102-492b-b5bb-e31e56e102b5\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") "
Jul 17 00:23:59.031430 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.031434    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd7f729-5102-492b-b5bb-e31e56e102b5-extensions-socket-volume\") pod \"0dd7f729-5102-492b-b5bb-e31e56e102b5\" (UID: \"0dd7f729-5102-492b-b5bb-e31e56e102b5\") "
Jul 17 00:23:59.031761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.031737    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd7f729-5102-492b-b5bb-e31e56e102b5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0dd7f729-5102-492b-b5bb-e31e56e102b5" (UID: "0dd7f729-5102-492b-b5bb-e31e56e102b5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:23:59.033591 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.033567    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd7f729-5102-492b-b5bb-e31e56e102b5-kube-api-access-zfs5p" (OuterVolumeSpecName: "kube-api-access-zfs5p") pod "0dd7f729-5102-492b-b5bb-e31e56e102b5" (UID: "0dd7f729-5102-492b-b5bb-e31e56e102b5"). InnerVolumeSpecName "kube-api-access-zfs5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:23:59.132419 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.132386    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd7f729-5102-492b-b5bb-e31e56e102b5-extensions-socket-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:59.132419 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.132412    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfs5p\" (UniqueName: \"kubernetes.io/projected/0dd7f729-5102-492b-b5bb-e31e56e102b5-kube-api-access-zfs5p\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:23:59.968314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.968266    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd7f729-5102-492b-b5bb-e31e56e102b5" path="/var/lib/kubelet/pods/0dd7f729-5102-492b-b5bb-e31e56e102b5/volumes"
Jul 17 00:23:59.971165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:23:59.971142    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:59.973020 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:59.972994    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="0dd7f729-5102-492b-b5bb-e31e56e102b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:23:59.979494 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:23:59.979469    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-9gxkp\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="0dd7f729-5102-492b-b5bb-e31e56e102b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9gxkp"
Jul 17 00:24:07.968293 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:07.968212    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:24:07.968293 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:07.968287    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:24:08.040210 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.040169    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"]
Jul 17 00:24:08.040435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.040404    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl" podUID="2d25088e-ff2a-470b-b536-9b11d4e9ef7f" containerName="manager" containerID="cri-o://ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669" gracePeriod=10
Jul 17 00:24:08.293861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.293832    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"]
Jul 17 00:24:08.294167 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.294155    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" containerName="console"
Jul 17 00:24:08.294212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.294168    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" containerName="console"
Jul 17 00:24:08.294253 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.294235    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eea02e1-aeb2-48eb-89c6-b6d11fb3b875" containerName="console"
Jul 17 00:24:08.296182 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.296160    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.297385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.297366    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:24:08.311533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.311504    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"]
Jul 17 00:24:08.410983 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.410942    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6jxd\" (UniqueName: \"kubernetes.io/projected/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-kube-api-access-h6jxd\") pod \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") "
Jul 17 00:24:08.411179 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.411001    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-extensions-socket-volume\") pod \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\" (UID: \"2d25088e-ff2a-470b-b536-9b11d4e9ef7f\") "
Jul 17 00:24:08.411179 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.411139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh795\" (UniqueName: \"kubernetes.io/projected/ab12ab81-15fa-47d8-ae19-d329af8229ff-kube-api-access-jh795\") pod \"kuadrant-operator-controller-manager-86878c769f-kdt7z\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.411294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.411201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab12ab81-15fa-47d8-ae19-d329af8229ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdt7z\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.411384 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.411360    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2d25088e-ff2a-470b-b536-9b11d4e9ef7f" (UID: "2d25088e-ff2a-470b-b536-9b11d4e9ef7f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:24:08.413139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.413106    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-kube-api-access-h6jxd" (OuterVolumeSpecName: "kube-api-access-h6jxd") pod "2d25088e-ff2a-470b-b536-9b11d4e9ef7f" (UID: "2d25088e-ff2a-470b-b536-9b11d4e9ef7f"). InnerVolumeSpecName "kube-api-access-h6jxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:24:08.511977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.511945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab12ab81-15fa-47d8-ae19-d329af8229ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdt7z\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.512139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.512033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh795\" (UniqueName: \"kubernetes.io/projected/ab12ab81-15fa-47d8-ae19-d329af8229ff-kube-api-access-jh795\") pod \"kuadrant-operator-controller-manager-86878c769f-kdt7z\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.512139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.512079    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6jxd\" (UniqueName: \"kubernetes.io/projected/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-kube-api-access-h6jxd\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:24:08.512139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.512097    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d25088e-ff2a-470b-b536-9b11d4e9ef7f-extensions-socket-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:24:08.512340 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.512322    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab12ab81-15fa-47d8-ae19-d329af8229ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdt7z\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.520960 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.520933    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh795\" (UniqueName: \"kubernetes.io/projected/ab12ab81-15fa-47d8-ae19-d329af8229ff-kube-api-access-jh795\") pod \"kuadrant-operator-controller-manager-86878c769f-kdt7z\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.605987 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.605946    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:08.746755 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:08.746725    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"]
Jul 17 00:24:08.749287 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:24:08.749261    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab12ab81_15fa_47d8_ae19_d329af8229ff.slice/crio-737496c2b43bb67dcfd5a3edcaf58054a1d5b119d1b4eab1c879c592b9742402 WatchSource:0}: Error finding container 737496c2b43bb67dcfd5a3edcaf58054a1d5b119d1b4eab1c879c592b9742402: Status 404 returned error can't find the container with id 737496c2b43bb67dcfd5a3edcaf58054a1d5b119d1b4eab1c879c592b9742402
Jul 17 00:24:09.022108 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.022015    2573 generic.go:356] "Generic (PLEG): container finished" podID="2d25088e-ff2a-470b-b536-9b11d4e9ef7f" containerID="ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669" exitCode=0
Jul 17 00:24:09.022108 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.022065    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"
Jul 17 00:24:09.022812 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.022103    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl" event={"ID":"2d25088e-ff2a-470b-b536-9b11d4e9ef7f","Type":"ContainerDied","Data":"ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669"}
Jul 17 00:24:09.022812 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.022156    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl" event={"ID":"2d25088e-ff2a-470b-b536-9b11d4e9ef7f","Type":"ContainerDied","Data":"7a590a302f774fb68d2a396e20a1cffcf8b622c167988a279c3ff228aafa88a3"}
Jul 17 00:24:09.022812 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.022192    2573 scope.go:117] "RemoveContainer" containerID="ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669"
Jul 17 00:24:09.023844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.023816    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z" event={"ID":"ab12ab81-15fa-47d8-ae19-d329af8229ff","Type":"ContainerStarted","Data":"0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5"}
Jul 17 00:24:09.023952 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.023876    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:09.023952 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.023901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z" event={"ID":"ab12ab81-15fa-47d8-ae19-d329af8229ff","Type":"ContainerStarted","Data":"737496c2b43bb67dcfd5a3edcaf58054a1d5b119d1b4eab1c879c592b9742402"}
Jul 17 00:24:09.031409 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.031390    2573 scope.go:117] "RemoveContainer" containerID="ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669"
Jul 17 00:24:09.031699 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:24:09.031677    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669\": container with ID starting with ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669 not found: ID does not exist" containerID="ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669"
Jul 17 00:24:09.031748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.031711    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669"} err="failed to get container status \"ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669\": rpc error: code = NotFound desc = could not find container \"ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669\": container with ID starting with ec0818bc542fbd3c769f40d716e0b0ddc11a037cda7e31e1dd36cfebe36f6669 not found: ID does not exist"
Jul 17 00:24:09.065830 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.065767    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"]
Jul 17 00:24:09.086533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.086494    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rt8xl"]
Jul 17 00:24:09.117209 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.117152    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z" podStartSLOduration=1.117135735 podStartE2EDuration="1.117135735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:24:09.114886354 +0000 UTC m=+443.691038927" watchObservedRunningTime="2026-07-17 00:24:09.117135735 +0000 UTC m=+443.693288310"
Jul 17 00:24:09.968302 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:09.968262    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d25088e-ff2a-470b-b536-9b11d4e9ef7f" path="/var/lib/kubelet/pods/2d25088e-ff2a-470b-b536-9b11d4e9ef7f/volumes"
Jul 17 00:24:20.032715 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:20.032675    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:24:20.083104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:20.083068    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"]
Jul 17 00:24:20.083510 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:20.083316    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv" podUID="01d6062e-fbd8-47a6-af89-f25326194254" containerName="manager" containerID="cri-o://0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320" gracePeriod=10
Jul 17 00:24:20.879480 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:20.879446    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:24:21.018324 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.018289    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5w9t\" (UniqueName: \"kubernetes.io/projected/01d6062e-fbd8-47a6-af89-f25326194254-kube-api-access-x5w9t\") pod \"01d6062e-fbd8-47a6-af89-f25326194254\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") "
Jul 17 00:24:21.018517 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.018393    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d6062e-fbd8-47a6-af89-f25326194254-extensions-socket-volume\") pod \"01d6062e-fbd8-47a6-af89-f25326194254\" (UID: \"01d6062e-fbd8-47a6-af89-f25326194254\") "
Jul 17 00:24:21.018903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.018869    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d6062e-fbd8-47a6-af89-f25326194254-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "01d6062e-fbd8-47a6-af89-f25326194254" (UID: "01d6062e-fbd8-47a6-af89-f25326194254"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:24:21.020681 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.020649    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d6062e-fbd8-47a6-af89-f25326194254-kube-api-access-x5w9t" (OuterVolumeSpecName: "kube-api-access-x5w9t") pod "01d6062e-fbd8-47a6-af89-f25326194254" (UID: "01d6062e-fbd8-47a6-af89-f25326194254"). InnerVolumeSpecName "kube-api-access-x5w9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:24:21.096932 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.096890    2573 generic.go:356] "Generic (PLEG): container finished" podID="01d6062e-fbd8-47a6-af89-f25326194254" containerID="0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320" exitCode=0
Jul 17 00:24:21.097404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.096948    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"
Jul 17 00:24:21.097404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.096971    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv" event={"ID":"01d6062e-fbd8-47a6-af89-f25326194254","Type":"ContainerDied","Data":"0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320"}
Jul 17 00:24:21.097404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.097050    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv" event={"ID":"01d6062e-fbd8-47a6-af89-f25326194254","Type":"ContainerDied","Data":"451b374a96c4cf2e4b78f475988703092c96c043f6397acbf267a3d03ab50468"}
Jul 17 00:24:21.097404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.097082    2573 scope.go:117] "RemoveContainer" containerID="0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320"
Jul 17 00:24:21.106917 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.106890    2573 scope.go:117] "RemoveContainer" containerID="0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320"
Jul 17 00:24:21.107240 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:24:21.107221    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320\": container with ID starting with 0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320 not found: ID does not exist" containerID="0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320"
Jul 17 00:24:21.107295 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.107252    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320"} err="failed to get container status \"0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320\": rpc error: code = NotFound desc = could not find container \"0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320\": container with ID starting with 0a7eb7a4dcb429a502aa087b10c320d786e0f23616804ffbf4b5decfd73d1320 not found: ID does not exist"
Jul 17 00:24:21.119206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.119175    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d6062e-fbd8-47a6-af89-f25326194254-extensions-socket-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:24:21.119206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.119204    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5w9t\" (UniqueName: \"kubernetes.io/projected/01d6062e-fbd8-47a6-af89-f25326194254-kube-api-access-x5w9t\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:24:21.123026 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.122987    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"]
Jul 17 00:24:21.126484 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.126454    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ks7mv"]
Jul 17 00:24:21.967747 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:21.967714    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d6062e-fbd8-47a6-af89-f25326194254" path="/var/lib/kubelet/pods/01d6062e-fbd8-47a6-af89-f25326194254/volumes"
Jul 17 00:24:39.897304 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.897267    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"]
Jul 17 00:24:39.897884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.897599    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d25088e-ff2a-470b-b536-9b11d4e9ef7f" containerName="manager"
Jul 17 00:24:39.897884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.897610    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d25088e-ff2a-470b-b536-9b11d4e9ef7f" containerName="manager"
Jul 17 00:24:39.897884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.897634    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d6062e-fbd8-47a6-af89-f25326194254" containerName="manager"
Jul 17 00:24:39.897884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.897640    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d6062e-fbd8-47a6-af89-f25326194254" containerName="manager"
Jul 17 00:24:39.897884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.897692    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d6062e-fbd8-47a6-af89-f25326194254" containerName="manager"
Jul 17 00:24:39.897884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.897699    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d25088e-ff2a-470b-b536-9b11d4e9ef7f" containerName="manager"
Jul 17 00:24:39.899676 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.899658    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:39.901567 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.901546    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 00:24:39.907637 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.907609    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"]
Jul 17 00:24:39.988226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.988186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvd92\" (UniqueName: \"kubernetes.io/projected/c926ae68-32c6-4856-88e4-8d950e12ee6a-kube-api-access-vvd92\") pod \"limitador-limitador-64c94fd64f-h5jtw\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:39.988418 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.988269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c926ae68-32c6-4856-88e4-8d950e12ee6a-config-file\") pod \"limitador-limitador-64c94fd64f-h5jtw\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:39.994087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:39.994053    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"]
Jul 17 00:24:40.089622 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.089580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c926ae68-32c6-4856-88e4-8d950e12ee6a-config-file\") pod \"limitador-limitador-64c94fd64f-h5jtw\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:40.089877 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.089685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd92\" (UniqueName: \"kubernetes.io/projected/c926ae68-32c6-4856-88e4-8d950e12ee6a-kube-api-access-vvd92\") pod \"limitador-limitador-64c94fd64f-h5jtw\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:40.090431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.090408    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c926ae68-32c6-4856-88e4-8d950e12ee6a-config-file\") pod \"limitador-limitador-64c94fd64f-h5jtw\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:40.098203 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.098170    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvd92\" (UniqueName: \"kubernetes.io/projected/c926ae68-32c6-4856-88e4-8d950e12ee6a-kube-api-access-vvd92\") pod \"limitador-limitador-64c94fd64f-h5jtw\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:40.211367 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.211274    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:40.372915 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.372883    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"]
Jul 17 00:24:40.375116 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:24:40.375086    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc926ae68_32c6_4856_88e4_8d950e12ee6a.slice/crio-f0e23b2ccd38bc3ddb6ecd0816fcf784cb3a52742573901912137c9cd914471d WatchSource:0}: Error finding container f0e23b2ccd38bc3ddb6ecd0816fcf784cb3a52742573901912137c9cd914471d: Status 404 returned error can't find the container with id f0e23b2ccd38bc3ddb6ecd0816fcf784cb3a52742573901912137c9cd914471d
Jul 17 00:24:40.682090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.682049    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wq8jm"]
Jul 17 00:24:40.685128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.685101    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wq8jm"
Jul 17 00:24:40.687074 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.687051    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4jrtd\""
Jul 17 00:24:40.691742 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.691711    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wq8jm"]
Jul 17 00:24:40.800971 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.800933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprzz\" (UniqueName: \"kubernetes.io/projected/7038fe56-5e1f-448f-adf6-123106a8da4b-kube-api-access-sprzz\") pod \"authorino-58cd955675-wq8jm\" (UID: \"7038fe56-5e1f-448f-adf6-123106a8da4b\") " pod="kuadrant-system/authorino-58cd955675-wq8jm"
Jul 17 00:24:40.901534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.901489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprzz\" (UniqueName: \"kubernetes.io/projected/7038fe56-5e1f-448f-adf6-123106a8da4b-kube-api-access-sprzz\") pod \"authorino-58cd955675-wq8jm\" (UID: \"7038fe56-5e1f-448f-adf6-123106a8da4b\") " pod="kuadrant-system/authorino-58cd955675-wq8jm"
Jul 17 00:24:40.909007 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.908976    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprzz\" (UniqueName: \"kubernetes.io/projected/7038fe56-5e1f-448f-adf6-123106a8da4b-kube-api-access-sprzz\") pod \"authorino-58cd955675-wq8jm\" (UID: \"7038fe56-5e1f-448f-adf6-123106a8da4b\") " pod="kuadrant-system/authorino-58cd955675-wq8jm"
Jul 17 00:24:40.997447 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:40.997358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wq8jm"
Jul 17 00:24:41.143749 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:41.143692    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wq8jm"]
Jul 17 00:24:41.150627 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:24:41.150590    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7038fe56_5e1f_448f_adf6_123106a8da4b.slice/crio-352cbcad0e63ccb6651b90d4d4e9c274c2e5b02aef4f92673faf84210c2e8421 WatchSource:0}: Error finding container 352cbcad0e63ccb6651b90d4d4e9c274c2e5b02aef4f92673faf84210c2e8421: Status 404 returned error can't find the container with id 352cbcad0e63ccb6651b90d4d4e9c274c2e5b02aef4f92673faf84210c2e8421
Jul 17 00:24:41.210823 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:41.210749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw" event={"ID":"c926ae68-32c6-4856-88e4-8d950e12ee6a","Type":"ContainerStarted","Data":"f0e23b2ccd38bc3ddb6ecd0816fcf784cb3a52742573901912137c9cd914471d"}
Jul 17 00:24:41.213385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:41.213299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wq8jm" event={"ID":"7038fe56-5e1f-448f-adf6-123106a8da4b","Type":"ContainerStarted","Data":"352cbcad0e63ccb6651b90d4d4e9c274c2e5b02aef4f92673faf84210c2e8421"}
Jul 17 00:24:44.330133 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:44.330093    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wq8jm"]
Jul 17 00:24:45.245189 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.245136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wq8jm" event={"ID":"7038fe56-5e1f-448f-adf6-123106a8da4b","Type":"ContainerStarted","Data":"a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646"}
Jul 17 00:24:45.245453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.245230    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wq8jm" podUID="7038fe56-5e1f-448f-adf6-123106a8da4b" containerName="authorino" containerID="cri-o://a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646" gracePeriod=30
Jul 17 00:24:45.247009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.246982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw" event={"ID":"c926ae68-32c6-4856-88e4-8d950e12ee6a","Type":"ContainerStarted","Data":"1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf"}
Jul 17 00:24:45.247125 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.247041    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:45.259062 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.259011    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wq8jm" podStartSLOduration=1.352258127 podStartE2EDuration="5.25899543s" totalImagesPullingTime="3.906737303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:24:45.257871227 +0000 UTC m=+479.834023802" watchObservedRunningTime="2026-07-17 00:24:45.25899543 +0000 UTC m=+479.835148004"
Jul 17 00:24:45.271525 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.271467    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw" podStartSLOduration=1.593296383 podStartE2EDuration="6.271446495s" totalImagesPullingTime="4.678150112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:24:45.271361901 +0000 UTC m=+479.847514476" watchObservedRunningTime="2026-07-17 00:24:45.271446495 +0000 UTC m=+479.847599069"
Jul 17 00:24:45.505943 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.505919    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wq8jm"
Jul 17 00:24:45.649299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.649191    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sprzz\" (UniqueName: \"kubernetes.io/projected/7038fe56-5e1f-448f-adf6-123106a8da4b-kube-api-access-sprzz\") pod \"7038fe56-5e1f-448f-adf6-123106a8da4b\" (UID: \"7038fe56-5e1f-448f-adf6-123106a8da4b\") "
Jul 17 00:24:45.651406 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.651380    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7038fe56-5e1f-448f-adf6-123106a8da4b-kube-api-access-sprzz" (OuterVolumeSpecName: "kube-api-access-sprzz") pod "7038fe56-5e1f-448f-adf6-123106a8da4b" (UID: "7038fe56-5e1f-448f-adf6-123106a8da4b"). InnerVolumeSpecName "kube-api-access-sprzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:24:45.750627 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:45.750570    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprzz\" (UniqueName: \"kubernetes.io/projected/7038fe56-5e1f-448f-adf6-123106a8da4b-kube-api-access-sprzz\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:24:46.254443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.254414    2573 generic.go:356] "Generic (PLEG): container finished" podID="7038fe56-5e1f-448f-adf6-123106a8da4b" containerID="a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646" exitCode=0
Jul 17 00:24:46.254598 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.254483    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wq8jm"
Jul 17 00:24:46.254598 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.254490    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wq8jm" event={"ID":"7038fe56-5e1f-448f-adf6-123106a8da4b","Type":"ContainerDied","Data":"a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646"}
Jul 17 00:24:46.254598 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.254534    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wq8jm" event={"ID":"7038fe56-5e1f-448f-adf6-123106a8da4b","Type":"ContainerDied","Data":"352cbcad0e63ccb6651b90d4d4e9c274c2e5b02aef4f92673faf84210c2e8421"}
Jul 17 00:24:46.254598 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.254557    2573 scope.go:117] "RemoveContainer" containerID="a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646"
Jul 17 00:24:46.263782 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.263760    2573 scope.go:117] "RemoveContainer" containerID="a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646"
Jul 17 00:24:46.264089 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:24:46.264072    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646\": container with ID starting with a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646 not found: ID does not exist" containerID="a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646"
Jul 17 00:24:46.264146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.264100    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646"} err="failed to get container status \"a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646\": rpc error: code = NotFound desc = could not find container \"a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646\": container with ID starting with a27632b6cd952d0b76ebb7b00c73a52104c33761981bc0ded832df28fcf9b646 not found: ID does not exist"
Jul 17 00:24:46.268806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.268765    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wq8jm"]
Jul 17 00:24:46.274665 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:46.274633    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wq8jm"]
Jul 17 00:24:47.968554 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:47.968513    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7038fe56-5e1f-448f-adf6-123106a8da4b" path="/var/lib/kubelet/pods/7038fe56-5e1f-448f-adf6-123106a8da4b/volumes"
Jul 17 00:24:54.231535 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.231495    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"]
Jul 17 00:24:54.232122 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.231754    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw" podUID="c926ae68-32c6-4856-88e4-8d950e12ee6a" containerName="limitador" containerID="cri-o://1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf" gracePeriod=30
Jul 17 00:24:54.232649 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.232553    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:54.794319 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.794237    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:54.934053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.934019    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c926ae68-32c6-4856-88e4-8d950e12ee6a-config-file\") pod \"c926ae68-32c6-4856-88e4-8d950e12ee6a\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") "
Jul 17 00:24:54.934237 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.934089    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvd92\" (UniqueName: \"kubernetes.io/projected/c926ae68-32c6-4856-88e4-8d950e12ee6a-kube-api-access-vvd92\") pod \"c926ae68-32c6-4856-88e4-8d950e12ee6a\" (UID: \"c926ae68-32c6-4856-88e4-8d950e12ee6a\") "
Jul 17 00:24:54.934409 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.934385    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c926ae68-32c6-4856-88e4-8d950e12ee6a-config-file" (OuterVolumeSpecName: "config-file") pod "c926ae68-32c6-4856-88e4-8d950e12ee6a" (UID: "c926ae68-32c6-4856-88e4-8d950e12ee6a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:24:54.936244 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:54.936211    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c926ae68-32c6-4856-88e4-8d950e12ee6a-kube-api-access-vvd92" (OuterVolumeSpecName: "kube-api-access-vvd92") pod "c926ae68-32c6-4856-88e4-8d950e12ee6a" (UID: "c926ae68-32c6-4856-88e4-8d950e12ee6a"). InnerVolumeSpecName "kube-api-access-vvd92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:24:55.034776 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.034737    2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c926ae68-32c6-4856-88e4-8d950e12ee6a-config-file\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:24:55.034776 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.034774    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvd92\" (UniqueName: \"kubernetes.io/projected/c926ae68-32c6-4856-88e4-8d950e12ee6a-kube-api-access-vvd92\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:24:55.305230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.305172    2573 generic.go:356] "Generic (PLEG): container finished" podID="c926ae68-32c6-4856-88e4-8d950e12ee6a" containerID="1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf" exitCode=0
Jul 17 00:24:55.306039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.305242    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"
Jul 17 00:24:55.306039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.305272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw" event={"ID":"c926ae68-32c6-4856-88e4-8d950e12ee6a","Type":"ContainerDied","Data":"1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf"}
Jul 17 00:24:55.306039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.305370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h5jtw" event={"ID":"c926ae68-32c6-4856-88e4-8d950e12ee6a","Type":"ContainerDied","Data":"f0e23b2ccd38bc3ddb6ecd0816fcf784cb3a52742573901912137c9cd914471d"}
Jul 17 00:24:55.306039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.305405    2573 scope.go:117] "RemoveContainer" containerID="1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf"
Jul 17 00:24:55.321464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.321444    2573 scope.go:117] "RemoveContainer" containerID="1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf"
Jul 17 00:24:55.321852 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:24:55.321827    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf\": container with ID starting with 1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf not found: ID does not exist" containerID="1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf"
Jul 17 00:24:55.321920 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.321866    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf"} err="failed to get container status \"1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf\": rpc error: code = NotFound desc = could not find container \"1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf\": container with ID starting with 1c0c2e87eacd71bafc74d6c17aab71897318eadb32109d2c661d8686ae9fccbf not found: ID does not exist"
Jul 17 00:24:55.330443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.330412    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"]
Jul 17 00:24:55.332468 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.332443    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h5jtw"]
Jul 17 00:24:55.975688 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:55.975641    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c926ae68-32c6-4856-88e4-8d950e12ee6a" path="/var/lib/kubelet/pods/c926ae68-32c6-4856-88e4-8d950e12ee6a/volumes"
Jul 17 00:24:57.997558 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:57.997517    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"]
Jul 17 00:24:57.997980 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:57.997927    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c926ae68-32c6-4856-88e4-8d950e12ee6a" containerName="limitador"
Jul 17 00:24:57.997980 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:57.997948    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c926ae68-32c6-4856-88e4-8d950e12ee6a" containerName="limitador"
Jul 17 00:24:57.997980 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:57.997964    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7038fe56-5e1f-448f-adf6-123106a8da4b" containerName="authorino"
Jul 17 00:24:57.997980 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:57.997973    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7038fe56-5e1f-448f-adf6-123106a8da4b" containerName="authorino"
Jul 17 00:24:57.998188 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:57.998072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7038fe56-5e1f-448f-adf6-123106a8da4b" containerName="authorino"
Jul 17 00:24:57.998188 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:57.998088    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c926ae68-32c6-4856-88e4-8d950e12ee6a" containerName="limitador"
Jul 17 00:24:58.001446 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.001423    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.004443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.004408    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 00:24:58.004443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.004421    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 00:24:58.005006 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.004991    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9qx5j\""
Jul 17 00:24:58.005056 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.004995    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 00:24:58.009286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.009262    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"]
Jul 17 00:24:58.161265 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.161227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mr98\" (UniqueName: \"kubernetes.io/projected/11c30936-0258-4e90-8759-e8e550709078-kube-api-access-8mr98\") pod \"postgres-69fb4594cf-9jbtn\" (UID: \"11c30936-0258-4e90-8759-e8e550709078\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.161265 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.161269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11c30936-0258-4e90-8759-e8e550709078-data\") pod \"postgres-69fb4594cf-9jbtn\" (UID: \"11c30936-0258-4e90-8759-e8e550709078\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.262566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.262523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr98\" (UniqueName: \"kubernetes.io/projected/11c30936-0258-4e90-8759-e8e550709078-kube-api-access-8mr98\") pod \"postgres-69fb4594cf-9jbtn\" (UID: \"11c30936-0258-4e90-8759-e8e550709078\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.262566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.262572    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11c30936-0258-4e90-8759-e8e550709078-data\") pod \"postgres-69fb4594cf-9jbtn\" (UID: \"11c30936-0258-4e90-8759-e8e550709078\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.262954 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.262938    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11c30936-0258-4e90-8759-e8e550709078-data\") pod \"postgres-69fb4594cf-9jbtn\" (UID: \"11c30936-0258-4e90-8759-e8e550709078\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.275246 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.275221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mr98\" (UniqueName: \"kubernetes.io/projected/11c30936-0258-4e90-8759-e8e550709078-kube-api-access-8mr98\") pod \"postgres-69fb4594cf-9jbtn\" (UID: \"11c30936-0258-4e90-8759-e8e550709078\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.314170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.314130    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:24:58.449130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:58.449102    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"]
Jul 17 00:24:58.452490 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:24:58.452462    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c30936_0258_4e90_8759_e8e550709078.slice/crio-7c862a62a407c6f06df19300bc9a0775c0bce4b75717c9666facdde8924b87a4 WatchSource:0}: Error finding container 7c862a62a407c6f06df19300bc9a0775c0bce4b75717c9666facdde8924b87a4: Status 404 returned error can't find the container with id 7c862a62a407c6f06df19300bc9a0775c0bce4b75717c9666facdde8924b87a4
Jul 17 00:24:59.330108 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:24:59.330045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn" event={"ID":"11c30936-0258-4e90-8759-e8e550709078","Type":"ContainerStarted","Data":"7c862a62a407c6f06df19300bc9a0775c0bce4b75717c9666facdde8924b87a4"}
Jul 17 00:25:04.361200 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:04.361154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn" event={"ID":"11c30936-0258-4e90-8759-e8e550709078","Type":"ContainerStarted","Data":"37d3cbe0872db7353675dabb203f2582e7df0665728e5a5db43a91c1cc9f5f56"}
Jul 17 00:25:04.361629 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:04.361368    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:25:04.376508 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:04.376452    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn" podStartSLOduration=2.51513725 podStartE2EDuration="7.376433768s" totalImagesPullingTime="4.861296518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:25:04.374109965 +0000 UTC m=+498.950262539" watchObservedRunningTime="2026-07-17 00:25:04.376433768 +0000 UTC m=+498.952586343"
Jul 17 00:25:10.397004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:10.396972    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9jbtn"
Jul 17 00:25:10.934263 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:10.934227    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-477sq"]
Jul 17 00:25:10.937426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:10.937401    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-477sq"
Jul 17 00:25:10.939221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:10.939197    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4jrtd\""
Jul 17 00:25:10.944128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:10.944101    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-477sq"]
Jul 17 00:25:10.975440 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:10.975409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxrw\" (UniqueName: \"kubernetes.io/projected/183a1e0e-85a6-472b-8b39-ccf57aac8656-kube-api-access-2lxrw\") pod \"authorino-5d4b67c446-477sq\" (UID: \"183a1e0e-85a6-472b-8b39-ccf57aac8656\") " pod="kuadrant-system/authorino-5d4b67c446-477sq"
Jul 17 00:25:11.076409 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.076378    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxrw\" (UniqueName: \"kubernetes.io/projected/183a1e0e-85a6-472b-8b39-ccf57aac8656-kube-api-access-2lxrw\") pod \"authorino-5d4b67c446-477sq\" (UID: \"183a1e0e-85a6-472b-8b39-ccf57aac8656\") " pod="kuadrant-system/authorino-5d4b67c446-477sq"
Jul 17 00:25:11.091575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.091553    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxrw\" (UniqueName: \"kubernetes.io/projected/183a1e0e-85a6-472b-8b39-ccf57aac8656-kube-api-access-2lxrw\") pod \"authorino-5d4b67c446-477sq\" (UID: \"183a1e0e-85a6-472b-8b39-ccf57aac8656\") " pod="kuadrant-system/authorino-5d4b67c446-477sq"
Jul 17 00:25:11.183820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.183769    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-477sq"]
Jul 17 00:25:11.184005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.183992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-477sq"
Jul 17 00:25:11.218346 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.212307    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6694694c57-crfx6"]
Jul 17 00:25:11.220041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.220008    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.222114 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.222090    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 17 00:25:11.222314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.222284    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6694694c57-crfx6"]
Jul 17 00:25:11.277451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.277407    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee35f97c-9add-4632-a47f-fabbdb6d2367-tls-cert\") pod \"authorino-6694694c57-crfx6\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") " pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.277597 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.277575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zcz\" (UniqueName: \"kubernetes.io/projected/ee35f97c-9add-4632-a47f-fabbdb6d2367-kube-api-access-46zcz\") pod \"authorino-6694694c57-crfx6\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") " pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.320389 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.320364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-477sq"]
Jul 17 00:25:11.322228 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:25:11.322196    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod183a1e0e_85a6_472b_8b39_ccf57aac8656.slice/crio-e0b7ad6cc083117dc4d3cec8675287230f0a58b868ab4f97c129359f4d9d14b3 WatchSource:0}: Error finding container e0b7ad6cc083117dc4d3cec8675287230f0a58b868ab4f97c129359f4d9d14b3: Status 404 returned error can't find the container with id e0b7ad6cc083117dc4d3cec8675287230f0a58b868ab4f97c129359f4d9d14b3
Jul 17 00:25:11.377985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.377956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zcz\" (UniqueName: \"kubernetes.io/projected/ee35f97c-9add-4632-a47f-fabbdb6d2367-kube-api-access-46zcz\") pod \"authorino-6694694c57-crfx6\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") " pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.378102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.377999    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee35f97c-9add-4632-a47f-fabbdb6d2367-tls-cert\") pod \"authorino-6694694c57-crfx6\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") " pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.380260 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.380238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee35f97c-9add-4632-a47f-fabbdb6d2367-tls-cert\") pod \"authorino-6694694c57-crfx6\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") " pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.385060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.385038    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zcz\" (UniqueName: \"kubernetes.io/projected/ee35f97c-9add-4632-a47f-fabbdb6d2367-kube-api-access-46zcz\") pod \"authorino-6694694c57-crfx6\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") " pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.398116 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.398088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-477sq" event={"ID":"183a1e0e-85a6-472b-8b39-ccf57aac8656","Type":"ContainerStarted","Data":"e0b7ad6cc083117dc4d3cec8675287230f0a58b868ab4f97c129359f4d9d14b3"}
Jul 17 00:25:11.532009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.531972    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:25:11.679915 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:11.679890    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6694694c57-crfx6"]
Jul 17 00:25:11.686912 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:25:11.686888    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee35f97c_9add_4632_a47f_fabbdb6d2367.slice/crio-ef07277090ce6d87b24d413344454cdf6ea8a360e914a03f7bc08ac2200dd2a3 WatchSource:0}: Error finding container ef07277090ce6d87b24d413344454cdf6ea8a360e914a03f7bc08ac2200dd2a3: Status 404 returned error can't find the container with id ef07277090ce6d87b24d413344454cdf6ea8a360e914a03f7bc08ac2200dd2a3
Jul 17 00:25:12.404456 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.404363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-477sq" event={"ID":"183a1e0e-85a6-472b-8b39-ccf57aac8656","Type":"ContainerStarted","Data":"7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9"}
Jul 17 00:25:12.404456 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.404378    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-477sq" podUID="183a1e0e-85a6-472b-8b39-ccf57aac8656" containerName="authorino" containerID="cri-o://7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9" gracePeriod=30
Jul 17 00:25:12.406020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.405992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6694694c57-crfx6" event={"ID":"ee35f97c-9add-4632-a47f-fabbdb6d2367","Type":"ContainerStarted","Data":"3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b"}
Jul 17 00:25:12.406130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.406036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6694694c57-crfx6" event={"ID":"ee35f97c-9add-4632-a47f-fabbdb6d2367","Type":"ContainerStarted","Data":"ef07277090ce6d87b24d413344454cdf6ea8a360e914a03f7bc08ac2200dd2a3"}
Jul 17 00:25:12.419003 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.418963    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-477sq" podStartSLOduration=2.086334439 podStartE2EDuration="2.418949208s" totalImagesPullingTime="332.614769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:25:12.417912063 +0000 UTC m=+506.994064649" watchObservedRunningTime="2026-07-17 00:25:12.418949208 +0000 UTC m=+506.995101782"
Jul 17 00:25:12.431042 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.431005    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6694694c57-crfx6" podStartSLOduration=1.09777597 podStartE2EDuration="1.430993327s" totalImagesPullingTime="333.217357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:25:12.430905322 +0000 UTC m=+507.007057897" watchObservedRunningTime="2026-07-17 00:25:12.430993327 +0000 UTC m=+507.007145900"
Jul 17 00:25:12.657354 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.657304    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-477sq"
Jul 17 00:25:12.689454 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.689432    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lxrw\" (UniqueName: \"kubernetes.io/projected/183a1e0e-85a6-472b-8b39-ccf57aac8656-kube-api-access-2lxrw\") pod \"183a1e0e-85a6-472b-8b39-ccf57aac8656\" (UID: \"183a1e0e-85a6-472b-8b39-ccf57aac8656\") "
Jul 17 00:25:12.691342 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.691316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183a1e0e-85a6-472b-8b39-ccf57aac8656-kube-api-access-2lxrw" (OuterVolumeSpecName: "kube-api-access-2lxrw") pod "183a1e0e-85a6-472b-8b39-ccf57aac8656" (UID: "183a1e0e-85a6-472b-8b39-ccf57aac8656"). InnerVolumeSpecName "kube-api-access-2lxrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:25:12.790557 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:12.790531    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lxrw\" (UniqueName: \"kubernetes.io/projected/183a1e0e-85a6-472b-8b39-ccf57aac8656-kube-api-access-2lxrw\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:25:13.413212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.413180    2573 generic.go:356] "Generic (PLEG): container finished" podID="183a1e0e-85a6-472b-8b39-ccf57aac8656" containerID="7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9" exitCode=0
Jul 17 00:25:13.413212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.413218    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-477sq"
Jul 17 00:25:13.413679 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.413258    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-477sq" event={"ID":"183a1e0e-85a6-472b-8b39-ccf57aac8656","Type":"ContainerDied","Data":"7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9"}
Jul 17 00:25:13.413679 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.413302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-477sq" event={"ID":"183a1e0e-85a6-472b-8b39-ccf57aac8656","Type":"ContainerDied","Data":"e0b7ad6cc083117dc4d3cec8675287230f0a58b868ab4f97c129359f4d9d14b3"}
Jul 17 00:25:13.413679 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.413323    2573 scope.go:117] "RemoveContainer" containerID="7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9"
Jul 17 00:25:13.421634 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.421618    2573 scope.go:117] "RemoveContainer" containerID="7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9"
Jul 17 00:25:13.421930 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:25:13.421912    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9\": container with ID starting with 7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9 not found: ID does not exist" containerID="7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9"
Jul 17 00:25:13.421996 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.421939    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9"} err="failed to get container status \"7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9\": rpc error: code = NotFound desc = could not find container \"7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9\": container with ID starting with 7ca26f31b4e4437465b3d0e2e7ca166ee4e92e4d8a7d05e49140f86de80e6ed9 not found: ID does not exist"
Jul 17 00:25:13.432320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.432297    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-477sq"]
Jul 17 00:25:13.434322 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.434303    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-477sq"]
Jul 17 00:25:13.967696 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:13.967666    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183a1e0e-85a6-472b-8b39-ccf57aac8656" path="/var/lib/kubelet/pods/183a1e0e-85a6-472b-8b39-ccf57aac8656/volumes"
Jul 17 00:25:16.992298 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:16.992265    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cdc9548dd-d5hg9"]
Jul 17 00:25:16.992707 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:16.992625    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183a1e0e-85a6-472b-8b39-ccf57aac8656" containerName="authorino"
Jul 17 00:25:16.992707 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:16.992638    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="183a1e0e-85a6-472b-8b39-ccf57aac8656" containerName="authorino"
Jul 17 00:25:16.992823 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:16.992723    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="183a1e0e-85a6-472b-8b39-ccf57aac8656" containerName="authorino"
Jul 17 00:25:17.166283 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.166249    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-d5hg9"]
Jul 17 00:25:17.166451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.166357    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.169038 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.169011    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jq5zc\""
Jul 17 00:25:17.169192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.169014    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 00:25:17.169192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.169018    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 00:25:17.230274 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.230240    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edca4e6f-9350-40fa-963c-40327d602b48-webhook-cert\") pod \"maas-controller-cdc9548dd-d5hg9\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") " pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.230448 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.230371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbq8p\" (UniqueName: \"kubernetes.io/projected/edca4e6f-9350-40fa-963c-40327d602b48-kube-api-access-pbq8p\") pod \"maas-controller-cdc9548dd-d5hg9\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") " pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.331169 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.331135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbq8p\" (UniqueName: \"kubernetes.io/projected/edca4e6f-9350-40fa-963c-40327d602b48-kube-api-access-pbq8p\") pod \"maas-controller-cdc9548dd-d5hg9\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") " pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.331169 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.331184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edca4e6f-9350-40fa-963c-40327d602b48-webhook-cert\") pod \"maas-controller-cdc9548dd-d5hg9\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") " pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.333630 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.333612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edca4e6f-9350-40fa-963c-40327d602b48-webhook-cert\") pod \"maas-controller-cdc9548dd-d5hg9\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") " pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.338700 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.338677    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbq8p\" (UniqueName: \"kubernetes.io/projected/edca4e6f-9350-40fa-963c-40327d602b48-kube-api-access-pbq8p\") pod \"maas-controller-cdc9548dd-d5hg9\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") " pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.477535 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.477503    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:17.647987 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:17.647964    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-d5hg9"]
Jul 17 00:25:17.649691 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:25:17.649663    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedca4e6f_9350_40fa_963c_40327d602b48.slice/crio-3643e8af2b4c86430cd3f84f79fe377682ef216ae63da17255185430658df8bf WatchSource:0}: Error finding container 3643e8af2b4c86430cd3f84f79fe377682ef216ae63da17255185430658df8bf: Status 404 returned error can't find the container with id 3643e8af2b4c86430cd3f84f79fe377682ef216ae63da17255185430658df8bf
Jul 17 00:25:18.442146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:18.442100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-d5hg9" event={"ID":"edca4e6f-9350-40fa-963c-40327d602b48","Type":"ContainerStarted","Data":"3643e8af2b4c86430cd3f84f79fe377682ef216ae63da17255185430658df8bf"}
Jul 17 00:25:20.454849 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:20.454736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-d5hg9" event={"ID":"edca4e6f-9350-40fa-963c-40327d602b48","Type":"ContainerStarted","Data":"4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5"}
Jul 17 00:25:20.455201 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:20.454847    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:20.477519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:20.477455    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cdc9548dd-d5hg9" podStartSLOduration=1.972536834 podStartE2EDuration="4.477436911s" totalImagesPullingTime="2.504900077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:25:20.473844214 +0000 UTC m=+515.049996784" watchObservedRunningTime="2026-07-17 00:25:20.477436911 +0000 UTC m=+515.053589505"
Jul 17 00:25:31.462684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.462657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:31.520720 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.520690    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-d5hg9"]
Jul 17 00:25:31.520902 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.520881    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cdc9548dd-d5hg9" podUID="edca4e6f-9350-40fa-963c-40327d602b48" containerName="manager" containerID="cri-o://4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5" gracePeriod=30
Jul 17 00:25:31.761572 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.761548    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:31.859289 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.859260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbq8p\" (UniqueName: \"kubernetes.io/projected/edca4e6f-9350-40fa-963c-40327d602b48-kube-api-access-pbq8p\") pod \"edca4e6f-9350-40fa-963c-40327d602b48\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") "
Jul 17 00:25:31.859425 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.859306    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edca4e6f-9350-40fa-963c-40327d602b48-webhook-cert\") pod \"edca4e6f-9350-40fa-963c-40327d602b48\" (UID: \"edca4e6f-9350-40fa-963c-40327d602b48\") "
Jul 17 00:25:31.861349 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.861321    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edca4e6f-9350-40fa-963c-40327d602b48-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "edca4e6f-9350-40fa-963c-40327d602b48" (UID: "edca4e6f-9350-40fa-963c-40327d602b48"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:25:31.861449 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.861390    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edca4e6f-9350-40fa-963c-40327d602b48-kube-api-access-pbq8p" (OuterVolumeSpecName: "kube-api-access-pbq8p") pod "edca4e6f-9350-40fa-963c-40327d602b48" (UID: "edca4e6f-9350-40fa-963c-40327d602b48"). InnerVolumeSpecName "kube-api-access-pbq8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:25:31.960422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.960393    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbq8p\" (UniqueName: \"kubernetes.io/projected/edca4e6f-9350-40fa-963c-40327d602b48-kube-api-access-pbq8p\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:25:31.960422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:31.960418    2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edca4e6f-9350-40fa-963c-40327d602b48-webhook-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:25:32.521061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.521028    2573 generic.go:356] "Generic (PLEG): container finished" podID="edca4e6f-9350-40fa-963c-40327d602b48" containerID="4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5" exitCode=0
Jul 17 00:25:32.521433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.521069    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-d5hg9"
Jul 17 00:25:32.521433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.521133    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-d5hg9" event={"ID":"edca4e6f-9350-40fa-963c-40327d602b48","Type":"ContainerDied","Data":"4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5"}
Jul 17 00:25:32.521433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.521179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-d5hg9" event={"ID":"edca4e6f-9350-40fa-963c-40327d602b48","Type":"ContainerDied","Data":"3643e8af2b4c86430cd3f84f79fe377682ef216ae63da17255185430658df8bf"}
Jul 17 00:25:32.521433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.521205    2573 scope.go:117] "RemoveContainer" containerID="4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5"
Jul 17 00:25:32.529674 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.529656    2573 scope.go:117] "RemoveContainer" containerID="4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5"
Jul 17 00:25:32.529950 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:25:32.529930    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5\": container with ID starting with 4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5 not found: ID does not exist" containerID="4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5"
Jul 17 00:25:32.530020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.529958    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5"} err="failed to get container status \"4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5\": rpc error: code = NotFound desc = could not find container \"4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5\": container with ID starting with 4e15f6112b8ef0957d9459be132ec6306055f449b8312fd743bd066f591b21f5 not found: ID does not exist"
Jul 17 00:25:32.536352 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.536331    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-d5hg9"]
Jul 17 00:25:32.541217 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:32.541194    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-d5hg9"]
Jul 17 00:25:33.967972 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:33.967930    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edca4e6f-9350-40fa-963c-40327d602b48" path="/var/lib/kubelet/pods/edca4e6f-9350-40fa-963c-40327d602b48/volumes"
Jul 17 00:25:53.519948 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.519907    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"]
Jul 17 00:25:53.520431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.520405    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edca4e6f-9350-40fa-963c-40327d602b48" containerName="manager"
Jul 17 00:25:53.520431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.520422    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="edca4e6f-9350-40fa-963c-40327d602b48" containerName="manager"
Jul 17 00:25:53.520562 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.520547    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="edca4e6f-9350-40fa-963c-40327d602b48" containerName="manager"
Jul 17 00:25:53.523434 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.523413    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.525752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.525725    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kjxwf\""
Jul 17 00:25:53.525877 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.525727    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 17 00:25:53.533896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.533874    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"]
Jul 17 00:25:53.613810 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.613753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b4198-05cf-4598-a670-096a6f7cf460-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pm4jz\" (UID: \"af6b4198-05cf-4598-a670-096a6f7cf460\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.613987 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.613848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktql\" (UniqueName: \"kubernetes.io/projected/af6b4198-05cf-4598-a670-096a6f7cf460-kube-api-access-kktql\") pod \"payload-pre-processing-7677b774bb-pm4jz\" (UID: \"af6b4198-05cf-4598-a670-096a6f7cf460\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.715081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.715047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b4198-05cf-4598-a670-096a6f7cf460-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pm4jz\" (UID: \"af6b4198-05cf-4598-a670-096a6f7cf460\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.715081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.715084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktql\" (UniqueName: \"kubernetes.io/projected/af6b4198-05cf-4598-a670-096a6f7cf460-kube-api-access-kktql\") pod \"payload-pre-processing-7677b774bb-pm4jz\" (UID: \"af6b4198-05cf-4598-a670-096a6f7cf460\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.715771 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.715750    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b4198-05cf-4598-a670-096a6f7cf460-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pm4jz\" (UID: \"af6b4198-05cf-4598-a670-096a6f7cf460\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.722183 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.722161    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktql\" (UniqueName: \"kubernetes.io/projected/af6b4198-05cf-4598-a670-096a6f7cf460-kube-api-access-kktql\") pod \"payload-pre-processing-7677b774bb-pm4jz\" (UID: \"af6b4198-05cf-4598-a670-096a6f7cf460\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.834860 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.834773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:53.976415 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:53.976385    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"]
Jul 17 00:25:53.977931 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:25:53.977907    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6b4198_05cf_4598_a670_096a6f7cf460.slice/crio-0bba3e282ac60a0cb3a65bde5983fbd8728a64bbc484a56ffb83cd9307b2e0fe WatchSource:0}: Error finding container 0bba3e282ac60a0cb3a65bde5983fbd8728a64bbc484a56ffb83cd9307b2e0fe: Status 404 returned error can't find the container with id 0bba3e282ac60a0cb3a65bde5983fbd8728a64bbc484a56ffb83cd9307b2e0fe
Jul 17 00:25:54.632112 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:54.632038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz" event={"ID":"af6b4198-05cf-4598-a670-096a6f7cf460","Type":"ContainerStarted","Data":"0bba3e282ac60a0cb3a65bde5983fbd8728a64bbc484a56ffb83cd9307b2e0fe"}
Jul 17 00:25:55.638317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:55.638280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz" event={"ID":"af6b4198-05cf-4598-a670-096a6f7cf460","Type":"ContainerStarted","Data":"5e541be199085d8da86a47af9e7b8fdac3a73b548b6ef401cf4ec2f63ab958e6"}
Jul 17 00:25:55.638679 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:55.638444    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:25:55.652188 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:25:55.652140    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz" podStartSLOduration=1.167406244 podStartE2EDuration="2.652123656s" totalImagesPullingTime="1.484717412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:25:55.650848758 +0000 UTC m=+550.227001323" watchObservedRunningTime="2026-07-17 00:25:55.652123656 +0000 UTC m=+550.228276231"
Jul 17 00:26:06.645196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:06.645166    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-pm4jz"
Jul 17 00:26:22.802636 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.802598    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"]
Jul 17 00:26:22.805940 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.805904    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.809164 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.809054    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 00:26:22.809164 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.809134    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 00:26:22.809164 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.809144    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nnsb6\""
Jul 17 00:26:22.809374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.809136    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 17 00:26:22.814868 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.814849    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"]
Jul 17 00:26:22.839712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.839686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.839831 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.839725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.839831 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.839751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.839831 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.839812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.839943 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.839833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.839943 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.839870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8dk\" (UniqueName: \"kubernetes.io/projected/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-kube-api-access-2x8dk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940468 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8dk\" (UniqueName: \"kubernetes.io/projected/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-kube-api-access-2x8dk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940634 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940609    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940654    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.940744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.940699    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.942595 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.942574    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.942812 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.942776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:22.947698 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:22.947680    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8dk\" (UniqueName: \"kubernetes.io/projected/cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba-kube-api-access-2x8dk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46\" (UID: \"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:23.117284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:23.117231    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:23.257162 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:23.257134    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"]
Jul 17 00:26:23.258885 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:26:23.258853    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5914f6_f2c7_4c76_8a4b_6bde1e6741ba.slice/crio-043cd789850a4fa9ac9e1daf9446bba50fb6f554db70231434fbd50e7d55dac4 WatchSource:0}: Error finding container 043cd789850a4fa9ac9e1daf9446bba50fb6f554db70231434fbd50e7d55dac4: Status 404 returned error can't find the container with id 043cd789850a4fa9ac9e1daf9446bba50fb6f554db70231434fbd50e7d55dac4
Jul 17 00:26:23.783891 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:23.783816    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46" event={"ID":"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba","Type":"ContainerStarted","Data":"043cd789850a4fa9ac9e1daf9446bba50fb6f554db70231434fbd50e7d55dac4"}
Jul 17 00:26:24.790093 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:24.790061    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46" event={"ID":"cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba","Type":"ContainerStarted","Data":"e2f4ac7aeeeeba3d7c5ac068815d5b1ee2d2a678669e3c7c2fd348d7709bc003"}
Jul 17 00:26:24.805273 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:24.805227    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46" podStartSLOduration=1.654855204 podStartE2EDuration="2.805210068s" totalImagesPullingTime="1.150354864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:26:24.805119395 +0000 UTC m=+579.381271968" watchObservedRunningTime="2026-07-17 00:26:24.805210068 +0000 UTC m=+579.381362645"
Jul 17 00:26:33.117911 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:33.117873    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:33.118297 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:33.117941    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:33.130195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:33.130171    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:33.849225 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:33.849198    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46"
Jul 17 00:26:35.197339 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.197303    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"]
Jul 17 00:26:35.202067 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.202046    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.204110 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.204094    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 00:26:35.211110 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.211090    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"]
Jul 17 00:26:35.231235 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.231216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.231322 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.231244    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.231322 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.231262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.231322 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.231278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8xz\" (UniqueName: \"kubernetes.io/projected/02c2e277-8128-4838-b7e5-b82baf799fce-kube-api-access-tm8xz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.231534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.231338    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.231534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.231362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c2e277-8128-4838-b7e5-b82baf799fce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.332664 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.332642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.332754 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.332675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c2e277-8128-4838-b7e5-b82baf799fce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.332848 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.332830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.332900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.332865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.332900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.332887    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.332985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.332910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8xz\" (UniqueName: \"kubernetes.io/projected/02c2e277-8128-4838-b7e5-b82baf799fce-kube-api-access-tm8xz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.333076 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.333045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.333178 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.333163    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.333231 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.333199    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.334834 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.334817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c2e277-8128-4838-b7e5-b82baf799fce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.335202 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.335183    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c2e277-8128-4838-b7e5-b82baf799fce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.340318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.340297    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8xz\" (UniqueName: \"kubernetes.io/projected/02c2e277-8128-4838-b7e5-b82baf799fce-kube-api-access-tm8xz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6zdk\" (UID: \"02c2e277-8128-4838-b7e5-b82baf799fce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.512315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.512291    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:35.637509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.637471    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"]
Jul 17 00:26:35.639063 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:26:35.639032    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c2e277_8128_4838_b7e5_b82baf799fce.slice/crio-d2a4215ffed0467d12b9aed9c7f5da1d23be524d3d7af5853027fabaaa127aab WatchSource:0}: Error finding container d2a4215ffed0467d12b9aed9c7f5da1d23be524d3d7af5853027fabaaa127aab: Status 404 returned error can't find the container with id d2a4215ffed0467d12b9aed9c7f5da1d23be524d3d7af5853027fabaaa127aab
Jul 17 00:26:35.850012 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:35.849985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk" event={"ID":"02c2e277-8128-4838-b7e5-b82baf799fce","Type":"ContainerStarted","Data":"d2a4215ffed0467d12b9aed9c7f5da1d23be524d3d7af5853027fabaaa127aab"}
Jul 17 00:26:36.856443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:36.856406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk" event={"ID":"02c2e277-8128-4838-b7e5-b82baf799fce","Type":"ContainerStarted","Data":"316082bc7b4c37a457b4eb21ff354447cd5cafaca1e1fff3c5c1d36f120d1174"}
Jul 17 00:26:36.874435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:36.874386    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk" podStartSLOduration=1.6906118399999999 podStartE2EDuration="1.874372126s" totalImagesPullingTime="183.760286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:26:36.8717653 +0000 UTC m=+591.447917872" watchObservedRunningTime="2026-07-17 00:26:36.874372126 +0000 UTC m=+591.450524698"
Jul 17 00:26:45.512895 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:45.512862    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:45.513246 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:45.512932    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:45.525058 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:45.525037    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:45.916752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:45.916678    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6zdk"
Jul 17 00:26:47.656462 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:47.656359    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:26:47.677565 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:26:47.658141    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:27:56.251278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.251244    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6694694c57-crfx6"]
Jul 17 00:27:56.251856 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.251457    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6694694c57-crfx6" podUID="ee35f97c-9add-4632-a47f-fabbdb6d2367" containerName="authorino" containerID="cri-o://3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b" gracePeriod=30
Jul 17 00:27:56.493151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.493130    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:27:56.539325 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.539296    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46zcz\" (UniqueName: \"kubernetes.io/projected/ee35f97c-9add-4632-a47f-fabbdb6d2367-kube-api-access-46zcz\") pod \"ee35f97c-9add-4632-a47f-fabbdb6d2367\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") "
Jul 17 00:27:56.539466 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.539335    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee35f97c-9add-4632-a47f-fabbdb6d2367-tls-cert\") pod \"ee35f97c-9add-4632-a47f-fabbdb6d2367\" (UID: \"ee35f97c-9add-4632-a47f-fabbdb6d2367\") "
Jul 17 00:27:56.541327 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.541297    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee35f97c-9add-4632-a47f-fabbdb6d2367-kube-api-access-46zcz" (OuterVolumeSpecName: "kube-api-access-46zcz") pod "ee35f97c-9add-4632-a47f-fabbdb6d2367" (UID: "ee35f97c-9add-4632-a47f-fabbdb6d2367"). InnerVolumeSpecName "kube-api-access-46zcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:27:56.550731 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.550706    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee35f97c-9add-4632-a47f-fabbdb6d2367-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ee35f97c-9add-4632-a47f-fabbdb6d2367" (UID: "ee35f97c-9add-4632-a47f-fabbdb6d2367"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:27:56.640401 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.640373    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zcz\" (UniqueName: \"kubernetes.io/projected/ee35f97c-9add-4632-a47f-fabbdb6d2367-kube-api-access-46zcz\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:27:56.640560 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:56.640403    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee35f97c-9add-4632-a47f-fabbdb6d2367-tls-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:27:57.264088 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.264055    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee35f97c-9add-4632-a47f-fabbdb6d2367" containerID="3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b" exitCode=0
Jul 17 00:27:57.264508 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.264093    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6694694c57-crfx6"
Jul 17 00:27:57.264508 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.264134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6694694c57-crfx6" event={"ID":"ee35f97c-9add-4632-a47f-fabbdb6d2367","Type":"ContainerDied","Data":"3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b"}
Jul 17 00:27:57.264508 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.264172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6694694c57-crfx6" event={"ID":"ee35f97c-9add-4632-a47f-fabbdb6d2367","Type":"ContainerDied","Data":"ef07277090ce6d87b24d413344454cdf6ea8a360e914a03f7bc08ac2200dd2a3"}
Jul 17 00:27:57.264508 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.264189    2573 scope.go:117] "RemoveContainer" containerID="3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b"
Jul 17 00:27:57.273893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.273814    2573 scope.go:117] "RemoveContainer" containerID="3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b"
Jul 17 00:27:57.274216 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:27:57.274153    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b\": container with ID starting with 3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b not found: ID does not exist" containerID="3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b"
Jul 17 00:27:57.274216 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.274185    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b"} err="failed to get container status \"3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b\": rpc error: code = NotFound desc = could not find container \"3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b\": container with ID starting with 3778867eadd8c077aab780e4c42c8c3db1110b22a7d2f9a2e5ddba608d677f8b not found: ID does not exist"
Jul 17 00:27:57.296434 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.296412    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6694694c57-crfx6"]
Jul 17 00:27:57.301888 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.301869    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6694694c57-crfx6"]
Jul 17 00:27:57.968166 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:27:57.968133    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee35f97c-9add-4632-a47f-fabbdb6d2367" path="/var/lib/kubelet/pods/ee35f97c-9add-4632-a47f-fabbdb6d2367/volumes"
Jul 17 00:29:44.553227 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.553194    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-965b4997-8gj82"]
Jul 17 00:29:44.553693 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.553518    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee35f97c-9add-4632-a47f-fabbdb6d2367" containerName="authorino"
Jul 17 00:29:44.553693 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.553529    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee35f97c-9add-4632-a47f-fabbdb6d2367" containerName="authorino"
Jul 17 00:29:44.553693 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.553592    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee35f97c-9add-4632-a47f-fabbdb6d2367" containerName="authorino"
Jul 17 00:29:44.556533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.556514    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.558621 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.558601    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jq5zc\""
Jul 17 00:29:44.559058 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.559039    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 00:29:44.559159 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.559125    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 00:29:44.565782 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.565759    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-965b4997-8gj82"]
Jul 17 00:29:44.690067 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.690036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c6898860-15d8-45b8-b039-b85d90ce21ec-webhook-cert\") pod \"maas-controller-965b4997-8gj82\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") " pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.690196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.690080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6kp\" (UniqueName: \"kubernetes.io/projected/c6898860-15d8-45b8-b039-b85d90ce21ec-kube-api-access-qp6kp\") pod \"maas-controller-965b4997-8gj82\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") " pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.790552 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.790523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c6898860-15d8-45b8-b039-b85d90ce21ec-webhook-cert\") pod \"maas-controller-965b4997-8gj82\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") " pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.790655 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.790565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6kp\" (UniqueName: \"kubernetes.io/projected/c6898860-15d8-45b8-b039-b85d90ce21ec-kube-api-access-qp6kp\") pod \"maas-controller-965b4997-8gj82\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") " pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.792782 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.792761    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c6898860-15d8-45b8-b039-b85d90ce21ec-webhook-cert\") pod \"maas-controller-965b4997-8gj82\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") " pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.797521 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.797497    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6kp\" (UniqueName: \"kubernetes.io/projected/c6898860-15d8-45b8-b039-b85d90ce21ec-kube-api-access-qp6kp\") pod \"maas-controller-965b4997-8gj82\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") " pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.868057 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.868009    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:44.997584 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.997554    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-965b4997-8gj82"]
Jul 17 00:29:44.998549 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:29:44.998515    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6898860_15d8_45b8_b039_b85d90ce21ec.slice/crio-cb79d7391697a89f32759e65037ac21c0b9432f06d16dff873cb71526f04868f WatchSource:0}: Error finding container cb79d7391697a89f32759e65037ac21c0b9432f06d16dff873cb71526f04868f: Status 404 returned error can't find the container with id cb79d7391697a89f32759e65037ac21c0b9432f06d16dff873cb71526f04868f
Jul 17 00:29:44.999897 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:44.999881    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 00:29:45.804908 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:45.804835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-965b4997-8gj82" event={"ID":"c6898860-15d8-45b8-b039-b85d90ce21ec","Type":"ContainerStarted","Data":"aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3"}
Jul 17 00:29:45.804908 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:45.804887    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-965b4997-8gj82" event={"ID":"c6898860-15d8-45b8-b039-b85d90ce21ec","Type":"ContainerStarted","Data":"cb79d7391697a89f32759e65037ac21c0b9432f06d16dff873cb71526f04868f"}
Jul 17 00:29:45.805276 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:45.805029    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:29:45.822080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:45.822035    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-965b4997-8gj82" podStartSLOduration=1.288752259 podStartE2EDuration="1.822021148s" totalImagesPullingTime="533.268889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:29:45.819237458 +0000 UTC m=+780.395390030" watchObservedRunningTime="2026-07-17 00:29:45.822021148 +0000 UTC m=+780.398173719"
Jul 17 00:29:56.812502 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:29:56.812473    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:30:00.140325 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:00.140287    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl"]
Jul 17 00:30:00.143822 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:00.143780    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl"
Jul 17 00:30:00.145887 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:00.145867    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:30:00.150409 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:00.150385    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl"]
Jul 17 00:30:00.155727 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:00.155708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl"
Jul 17 00:30:00.286763 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:00.286733    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl"]
Jul 17 00:30:00.289978 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:30:00.289950    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982d352b_b348_4ef2_b927_7bdeb7f4c3a6.slice/crio-80d3822abf477ea8c5d672af6f5bbea9ef3752e9ef76bebc8970b0a56da2880d WatchSource:0}: Error finding container 80d3822abf477ea8c5d672af6f5bbea9ef3752e9ef76bebc8970b0a56da2880d: Status 404 returned error can't find the container with id 80d3822abf477ea8c5d672af6f5bbea9ef3752e9ef76bebc8970b0a56da2880d
Jul 17 00:30:00.883995 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:00.883960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl" event={"ID":"982d352b-b348-4ef2-b927-7bdeb7f4c3a6","Type":"ContainerStarted","Data":"80d3822abf477ea8c5d672af6f5bbea9ef3752e9ef76bebc8970b0a56da2880d"}
Jul 17 00:30:01.397998 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.397777    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-965b4997-8gj82"]
Jul 17 00:30:01.398752 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.398232    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-965b4997-8gj82" podUID="c6898860-15d8-45b8-b039-b85d90ce21ec" containerName="manager" containerID="cri-o://aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3" gracePeriod=30
Jul 17 00:30:01.654805 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.654675    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:30:01.722467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.722441    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp6kp\" (UniqueName: \"kubernetes.io/projected/c6898860-15d8-45b8-b039-b85d90ce21ec-kube-api-access-qp6kp\") pod \"c6898860-15d8-45b8-b039-b85d90ce21ec\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") "
Jul 17 00:30:01.722639 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.722621    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c6898860-15d8-45b8-b039-b85d90ce21ec-webhook-cert\") pod \"c6898860-15d8-45b8-b039-b85d90ce21ec\" (UID: \"c6898860-15d8-45b8-b039-b85d90ce21ec\") "
Jul 17 00:30:01.724500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.724452    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6898860-15d8-45b8-b039-b85d90ce21ec-kube-api-access-qp6kp" (OuterVolumeSpecName: "kube-api-access-qp6kp") pod "c6898860-15d8-45b8-b039-b85d90ce21ec" (UID: "c6898860-15d8-45b8-b039-b85d90ce21ec"). InnerVolumeSpecName "kube-api-access-qp6kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:30:01.724618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.724553    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6898860-15d8-45b8-b039-b85d90ce21ec-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c6898860-15d8-45b8-b039-b85d90ce21ec" (UID: "c6898860-15d8-45b8-b039-b85d90ce21ec"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:30:01.823286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.823253    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp6kp\" (UniqueName: \"kubernetes.io/projected/c6898860-15d8-45b8-b039-b85d90ce21ec-kube-api-access-qp6kp\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:30:01.823286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.823284    2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c6898860-15d8-45b8-b039-b85d90ce21ec-webhook-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:30:01.890461 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.890422    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6898860-15d8-45b8-b039-b85d90ce21ec" containerID="aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3" exitCode=0
Jul 17 00:30:01.890644 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.890463    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-965b4997-8gj82"
Jul 17 00:30:01.890644 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.890501    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-965b4997-8gj82" event={"ID":"c6898860-15d8-45b8-b039-b85d90ce21ec","Type":"ContainerDied","Data":"aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3"}
Jul 17 00:30:01.890644 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.890553    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-965b4997-8gj82" event={"ID":"c6898860-15d8-45b8-b039-b85d90ce21ec","Type":"ContainerDied","Data":"cb79d7391697a89f32759e65037ac21c0b9432f06d16dff873cb71526f04868f"}
Jul 17 00:30:01.890644 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.890585    2573 scope.go:117] "RemoveContainer" containerID="aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3"
Jul 17 00:30:01.912245 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.912189    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-965b4997-8gj82"]
Jul 17 00:30:01.915642 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.915617    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-965b4997-8gj82"]
Jul 17 00:30:01.968928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:01.968897    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6898860-15d8-45b8-b039-b85d90ce21ec" path="/var/lib/kubelet/pods/c6898860-15d8-45b8-b039-b85d90ce21ec/volumes"
Jul 17 00:30:02.379460 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.379440    2573 scope.go:117] "RemoveContainer" containerID="aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3"
Jul 17 00:30:02.379746 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:30:02.379727    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3\": container with ID starting with aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3 not found: ID does not exist" containerID="aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3"
Jul 17 00:30:02.379820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.379756    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3"} err="failed to get container status \"aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3\": rpc error: code = NotFound desc = could not find container \"aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3\": container with ID starting with aa5506edfdaa1e63af82a595019747631f33bacd46f826de31c695c3c9b7e0d3 not found: ID does not exist"
Jul 17 00:30:02.804068 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.804033    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-965b4997-zn7mh"]
Jul 17 00:30:02.804467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.804369    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6898860-15d8-45b8-b039-b85d90ce21ec" containerName="manager"
Jul 17 00:30:02.804467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.804383    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6898860-15d8-45b8-b039-b85d90ce21ec" containerName="manager"
Jul 17 00:30:02.804467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.804454    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6898860-15d8-45b8-b039-b85d90ce21ec" containerName="manager"
Jul 17 00:30:02.807356 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.807340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:02.809697 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.809675    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 00:30:02.810337 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.810319    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 00:30:02.810410 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.810374    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jq5zc\""
Jul 17 00:30:02.829563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.829536    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-965b4997-zn7mh"]
Jul 17 00:30:02.829923 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.829897    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a21cd433-0636-49ed-871e-67a383ac3c87-webhook-cert\") pod \"maas-controller-965b4997-zn7mh\" (UID: \"a21cd433-0636-49ed-871e-67a383ac3c87\") " pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:02.829998 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.829955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dskz\" (UniqueName: \"kubernetes.io/projected/a21cd433-0636-49ed-871e-67a383ac3c87-kube-api-access-5dskz\") pod \"maas-controller-965b4997-zn7mh\" (UID: \"a21cd433-0636-49ed-871e-67a383ac3c87\") " pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:02.903763 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.903731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl" event={"ID":"982d352b-b348-4ef2-b927-7bdeb7f4c3a6","Type":"ContainerStarted","Data":"8227111542b23f0c56769c8b72c516a4eba648298b5121b0926e05d1112ae7d5"}
Jul 17 00:30:02.919510 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.919427    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl" podStartSLOduration=0.772871747 podStartE2EDuration="2.919412713s" totalImagesPullingTime="2.146540966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:30:02.917508672 +0000 UTC m=+797.493661247" watchObservedRunningTime="2026-07-17 00:30:02.919412713 +0000 UTC m=+797.495565285"
Jul 17 00:30:02.930757 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.930736    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dskz\" (UniqueName: \"kubernetes.io/projected/a21cd433-0636-49ed-871e-67a383ac3c87-kube-api-access-5dskz\") pod \"maas-controller-965b4997-zn7mh\" (UID: \"a21cd433-0636-49ed-871e-67a383ac3c87\") " pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:02.930862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.930847    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a21cd433-0636-49ed-871e-67a383ac3c87-webhook-cert\") pod \"maas-controller-965b4997-zn7mh\" (UID: \"a21cd433-0636-49ed-871e-67a383ac3c87\") " pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:02.933173 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.933156    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a21cd433-0636-49ed-871e-67a383ac3c87-webhook-cert\") pod \"maas-controller-965b4997-zn7mh\" (UID: \"a21cd433-0636-49ed-871e-67a383ac3c87\") " pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:02.938240 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:02.938215    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dskz\" (UniqueName: \"kubernetes.io/projected/a21cd433-0636-49ed-871e-67a383ac3c87-kube-api-access-5dskz\") pod \"maas-controller-965b4997-zn7mh\" (UID: \"a21cd433-0636-49ed-871e-67a383ac3c87\") " pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:03.117288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.117255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:03.244585 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.244555    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-965b4997-zn7mh"]
Jul 17 00:30:03.247619 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:30:03.247590    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21cd433_0636_49ed_871e_67a383ac3c87.slice/crio-2fe2fd831034dcc2ef12909cbbd22225687c9e3a8d9a0bb14df91fa1c5a2facc WatchSource:0}: Error finding container 2fe2fd831034dcc2ef12909cbbd22225687c9e3a8d9a0bb14df91fa1c5a2facc: Status 404 returned error can't find the container with id 2fe2fd831034dcc2ef12909cbbd22225687c9e3a8d9a0bb14df91fa1c5a2facc
Jul 17 00:30:03.910170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.910079    2573 generic.go:356] "Generic (PLEG): container finished" podID="982d352b-b348-4ef2-b927-7bdeb7f4c3a6" containerID="8227111542b23f0c56769c8b72c516a4eba648298b5121b0926e05d1112ae7d5" exitCode=0
Jul 17 00:30:03.910607 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.910161    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl" event={"ID":"982d352b-b348-4ef2-b927-7bdeb7f4c3a6","Type":"ContainerDied","Data":"8227111542b23f0c56769c8b72c516a4eba648298b5121b0926e05d1112ae7d5"}
Jul 17 00:30:03.911640 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.911616    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-965b4997-zn7mh" event={"ID":"a21cd433-0636-49ed-871e-67a383ac3c87","Type":"ContainerStarted","Data":"d5fa16a444fcd72a2d730f4b6e79b2af21d49570a8ee136a52ba2c4064150249"}
Jul 17 00:30:03.911708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.911670    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:30:03.911708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.911687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-965b4997-zn7mh" event={"ID":"a21cd433-0636-49ed-871e-67a383ac3c87","Type":"ContainerStarted","Data":"2fe2fd831034dcc2ef12909cbbd22225687c9e3a8d9a0bb14df91fa1c5a2facc"}
Jul 17 00:30:03.946405 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:03.946345    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-965b4997-zn7mh" podStartSLOduration=1.5077791 podStartE2EDuration="1.946325173s" totalImagesPullingTime="438.546073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:30:03.944355573 +0000 UTC m=+798.520508170" watchObservedRunningTime="2026-07-17 00:30:03.946325173 +0000 UTC m=+798.522477746"
Jul 17 00:30:05.042301 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:05.042274    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl"
Jul 17 00:30:05.923081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:05.923052    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl"
Jul 17 00:30:05.923081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:05.923078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737470-cq2gl" event={"ID":"982d352b-b348-4ef2-b927-7bdeb7f4c3a6","Type":"ContainerDied","Data":"80d3822abf477ea8c5d672af6f5bbea9ef3752e9ef76bebc8970b0a56da2880d"}
Jul 17 00:30:05.923289 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:05.923123    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d3822abf477ea8c5d672af6f5bbea9ef3752e9ef76bebc8970b0a56da2880d"
Jul 17 00:30:14.918840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:30:14.918750    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-965b4997-zn7mh"
Jul 17 00:31:47.687312 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:31:47.687205    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:31:47.689678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:31:47.689659    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:34:53.191944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:53.191915    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"]
Jul 17 00:34:53.192488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:53.192213    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z" podUID="ab12ab81-15fa-47d8-ae19-d329af8229ff" containerName="manager" containerID="cri-o://0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5" gracePeriod=10
Jul 17 00:34:56.339663 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.339640    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:34:56.423847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.423816    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab12ab81-15fa-47d8-ae19-d329af8229ff" containerID="0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5" exitCode=0
Jul 17 00:34:56.423992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.423862    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"
Jul 17 00:34:56.423992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.423906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z" event={"ID":"ab12ab81-15fa-47d8-ae19-d329af8229ff","Type":"ContainerDied","Data":"0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5"}
Jul 17 00:34:56.423992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.423960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z" event={"ID":"ab12ab81-15fa-47d8-ae19-d329af8229ff","Type":"ContainerDied","Data":"737496c2b43bb67dcfd5a3edcaf58054a1d5b119d1b4eab1c879c592b9742402"}
Jul 17 00:34:56.423992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.423983    2573 scope.go:117] "RemoveContainer" containerID="0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5"
Jul 17 00:34:56.433175 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.433160    2573 scope.go:117] "RemoveContainer" containerID="0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5"
Jul 17 00:34:56.433405 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:34:56.433387    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5\": container with ID starting with 0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5 not found: ID does not exist" containerID="0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5"
Jul 17 00:34:56.433465 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.433412    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5"} err="failed to get container status \"0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5\": rpc error: code = NotFound desc = could not find container \"0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5\": container with ID starting with 0308a186185a6f9440ff6d4decbc6d661ea76e89882a5387a136815f30848bd5 not found: ID does not exist"
Jul 17 00:34:56.474080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.474027    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab12ab81-15fa-47d8-ae19-d329af8229ff-extensions-socket-volume\") pod \"ab12ab81-15fa-47d8-ae19-d329af8229ff\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") "
Jul 17 00:34:56.474184 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.474089    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh795\" (UniqueName: \"kubernetes.io/projected/ab12ab81-15fa-47d8-ae19-d329af8229ff-kube-api-access-jh795\") pod \"ab12ab81-15fa-47d8-ae19-d329af8229ff\" (UID: \"ab12ab81-15fa-47d8-ae19-d329af8229ff\") "
Jul 17 00:34:56.474355 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.474335    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab12ab81-15fa-47d8-ae19-d329af8229ff-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ab12ab81-15fa-47d8-ae19-d329af8229ff" (UID: "ab12ab81-15fa-47d8-ae19-d329af8229ff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 00:34:56.475942 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.475919    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab12ab81-15fa-47d8-ae19-d329af8229ff-kube-api-access-jh795" (OuterVolumeSpecName: "kube-api-access-jh795") pod "ab12ab81-15fa-47d8-ae19-d329af8229ff" (UID: "ab12ab81-15fa-47d8-ae19-d329af8229ff"). InnerVolumeSpecName "kube-api-access-jh795". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:34:56.575046 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.575022    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab12ab81-15fa-47d8-ae19-d329af8229ff-extensions-socket-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:34:56.575046 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.575044    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh795\" (UniqueName: \"kubernetes.io/projected/ab12ab81-15fa-47d8-ae19-d329af8229ff-kube-api-access-jh795\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:34:56.744064 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.744031    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"]
Jul 17 00:34:56.747974 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:56.747938    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdt7z"]
Jul 17 00:34:57.968389 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:34:57.968355    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab12ab81-15fa-47d8-ae19-d329af8229ff" path="/var/lib/kubelet/pods/ab12ab81-15fa-47d8-ae19-d329af8229ff/volumes"
Jul 17 00:35:54.296663 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.296629    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"]
Jul 17 00:35:54.297284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.296974    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab12ab81-15fa-47d8-ae19-d329af8229ff" containerName="manager"
Jul 17 00:35:54.297284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.296986    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab12ab81-15fa-47d8-ae19-d329af8229ff" containerName="manager"
Jul 17 00:35:54.297284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.296997    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982d352b-b348-4ef2-b927-7bdeb7f4c3a6" containerName="cleanup"
Jul 17 00:35:54.297284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.297002    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="982d352b-b348-4ef2-b927-7bdeb7f4c3a6" containerName="cleanup"
Jul 17 00:35:54.297284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.297054    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab12ab81-15fa-47d8-ae19-d329af8229ff" containerName="manager"
Jul 17 00:35:54.297284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.297061    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="982d352b-b348-4ef2-b927-7bdeb7f4c3a6" containerName="cleanup"
Jul 17 00:35:54.299806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.299774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.301811 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.301771    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jg6jl\""
Jul 17 00:35:54.313691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.313669    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"]
Jul 17 00:35:54.418366 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.418338    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45577182-8092-4ac7-b438-cbb85b50724a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bjvkl\" (UID: \"45577182-8092-4ac7-b438-cbb85b50724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.418486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.418425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk7q\" (UniqueName: \"kubernetes.io/projected/45577182-8092-4ac7-b438-cbb85b50724a-kube-api-access-5lk7q\") pod \"kuadrant-operator-controller-manager-86878c769f-bjvkl\" (UID: \"45577182-8092-4ac7-b438-cbb85b50724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.518918 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.518892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45577182-8092-4ac7-b438-cbb85b50724a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bjvkl\" (UID: \"45577182-8092-4ac7-b438-cbb85b50724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.519033 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.519015    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk7q\" (UniqueName: \"kubernetes.io/projected/45577182-8092-4ac7-b438-cbb85b50724a-kube-api-access-5lk7q\") pod \"kuadrant-operator-controller-manager-86878c769f-bjvkl\" (UID: \"45577182-8092-4ac7-b438-cbb85b50724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.519251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.519236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45577182-8092-4ac7-b438-cbb85b50724a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bjvkl\" (UID: \"45577182-8092-4ac7-b438-cbb85b50724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.528420 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.528395    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk7q\" (UniqueName: \"kubernetes.io/projected/45577182-8092-4ac7-b438-cbb85b50724a-kube-api-access-5lk7q\") pod \"kuadrant-operator-controller-manager-86878c769f-bjvkl\" (UID: \"45577182-8092-4ac7-b438-cbb85b50724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.609950 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.609898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:54.740737 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:54.740716    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"]
Jul 17 00:35:54.742508 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:35:54.742484    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45577182_8092_4ac7_b438_cbb85b50724a.slice/crio-01e3468e27f877fca6e5478b3e08d6c5f5715fa17940370091c4eb888eb9ca29 WatchSource:0}: Error finding container 01e3468e27f877fca6e5478b3e08d6c5f5715fa17940370091c4eb888eb9ca29: Status 404 returned error can't find the container with id 01e3468e27f877fca6e5478b3e08d6c5f5715fa17940370091c4eb888eb9ca29
Jul 17 00:35:55.719019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:55.718985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl" event={"ID":"45577182-8092-4ac7-b438-cbb85b50724a","Type":"ContainerStarted","Data":"43e6e9ff88a8d427463922a7a07c4965cb6cfa55c03156a0151efd5f759ea690"}
Jul 17 00:35:55.719411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:55.719037    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl" event={"ID":"45577182-8092-4ac7-b438-cbb85b50724a","Type":"ContainerStarted","Data":"01e3468e27f877fca6e5478b3e08d6c5f5715fa17940370091c4eb888eb9ca29"}
Jul 17 00:35:55.719411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:55.719096    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:35:55.741159 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:35:55.741121    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl" podStartSLOduration=1.741108275 podStartE2EDuration="1.741108275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:35:55.738299066 +0000 UTC m=+1150.314451638" watchObservedRunningTime="2026-07-17 00:35:55.741108275 +0000 UTC m=+1150.317260847"
Jul 17 00:36:06.725905 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:36:06.725825    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bjvkl"
Jul 17 00:36:47.719905 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:36:47.719782    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:36:47.724132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:36:47.722921    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:41:47.758655 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:41:47.758630    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:41:47.763719 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:41:47.763701    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:42:07.576160 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.576123    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"]
Jul 17 00:42:07.579493 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.579469    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.582022 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.582005    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 00:42:07.582381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.582362    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b7aa65f0-serving-cert\""
Jul 17 00:42:07.600786 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.600766    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"]
Jul 17 00:42:07.682916 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.682880    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzdh9\" (UniqueName: \"kubernetes.io/projected/5b77c73a-88ae-4de3-bb81-66f573c468b0-kube-api-access-zzdh9\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.683054 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.683006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.683054 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.683048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b77c73a-88ae-4de3-bb81-66f573c468b0-openshift-service-ca\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.783594 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.783558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.783741 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.783615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b77c73a-88ae-4de3-bb81-66f573c468b0-openshift-service-ca\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.783741 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.783647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzdh9\" (UniqueName: \"kubernetes.io/projected/5b77c73a-88ae-4de3-bb81-66f573c468b0-kube-api-access-zzdh9\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.783741 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:07.783706    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-serving-cert: secret "maas-api-e2e-ait-b7aa65f0-serving-cert" not found
Jul 17 00:42:07.783951 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:07.783765    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls podName:5b77c73a-88ae-4de3-bb81-66f573c468b0 nodeName:}" failed. No retries permitted until 2026-07-17 00:42:08.283748042 +0000 UTC m=+1522.859900594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls") pod "maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" (UID: "5b77c73a-88ae-4de3-bb81-66f573c468b0") : secret "maas-api-e2e-ait-b7aa65f0-serving-cert" not found
Jul 17 00:42:07.784260 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.784241    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b77c73a-88ae-4de3-bb81-66f573c468b0-openshift-service-ca\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:07.792170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:07.792147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzdh9\" (UniqueName: \"kubernetes.io/projected/5b77c73a-88ae-4de3-bb81-66f573c468b0-kube-api-access-zzdh9\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:08.287206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:08.287167    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:08.289851 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:08.289825    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls\") pod \"maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:08.489410 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:08.489379    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:08.621081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:08.621052    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"]
Jul 17 00:42:08.622755 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:42:08.622730    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b77c73a_88ae_4de3_bb81_66f573c468b0.slice/crio-6d69d29cddc563690e26a22f699cd5312cc0078b8ef75a92ed546c99ada36a33 WatchSource:0}: Error finding container 6d69d29cddc563690e26a22f699cd5312cc0078b8ef75a92ed546c99ada36a33: Status 404 returned error can't find the container with id 6d69d29cddc563690e26a22f699cd5312cc0078b8ef75a92ed546c99ada36a33
Jul 17 00:42:08.623905 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:08.623887    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 00:42:09.630454 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:09.630413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerStarted","Data":"6d69d29cddc563690e26a22f699cd5312cc0078b8ef75a92ed546c99ada36a33"}
Jul 17 00:42:10.580773 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:10.580745    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b77c73a_88ae_4de3_bb81_66f573c468b0.slice/crio-84603126a53f2efadd7d86036de39a4ed616e9388a6aeb79b26d782fac54bf0b.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 00:42:10.636083 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:10.636052    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerID="84603126a53f2efadd7d86036de39a4ed616e9388a6aeb79b26d782fac54bf0b" exitCode=1
Jul 17 00:42:10.636355 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:10.636137    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerDied","Data":"84603126a53f2efadd7d86036de39a4ed616e9388a6aeb79b26d782fac54bf0b"}
Jul 17 00:42:10.636355 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:10.636315    2573 scope.go:117] "RemoveContainer" containerID="84603126a53f2efadd7d86036de39a4ed616e9388a6aeb79b26d782fac54bf0b"
Jul 17 00:42:12.302864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.302841    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"]
Jul 17 00:42:12.305990 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.305974    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"
Jul 17 00:42:12.307631 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.307616    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:42:12.312404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.312379    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"]
Jul 17 00:42:12.424077 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.424048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b888088-01c1-4c7e-971d-320ce004b459-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md\" (UID: \"0b888088-01c1-4c7e-971d-320ce004b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"
Jul 17 00:42:12.525446 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.525418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b888088-01c1-4c7e-971d-320ce004b459-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md\" (UID: \"0b888088-01c1-4c7e-971d-320ce004b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"
Jul 17 00:42:12.526066 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.526049    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b888088-01c1-4c7e-971d-320ce004b459-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md\" (UID: \"0b888088-01c1-4c7e-971d-320ce004b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"
Jul 17 00:42:12.617476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.617452    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"
Jul 17 00:42:12.648839 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.648785    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerID="b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f" exitCode=1
Jul 17 00:42:12.648839 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.648823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerDied","Data":"b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f"}
Jul 17 00:42:12.649001 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.648892    2573 scope.go:117] "RemoveContainer" containerID="84603126a53f2efadd7d86036de39a4ed616e9388a6aeb79b26d782fac54bf0b"
Jul 17 00:42:12.649095 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.649075    2573 scope.go:117] "RemoveContainer" containerID="b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f"
Jul 17 00:42:12.649305 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:12.649282    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:42:12.759667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:12.759596    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"]
Jul 17 00:42:12.760592 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:42:12.760551    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b888088_01c1_4c7e_971d_320ce004b459.slice/crio-86e2ec7f3491aebcc1b988c494850551a2628ede56e9a385755600135058c101 WatchSource:0}: Error finding container 86e2ec7f3491aebcc1b988c494850551a2628ede56e9a385755600135058c101: Status 404 returned error can't find the container with id 86e2ec7f3491aebcc1b988c494850551a2628ede56e9a385755600135058c101
Jul 17 00:42:13.490511 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:13.490476    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:13.656326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:13.656240    2573 scope.go:117] "RemoveContainer" containerID="b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f"
Jul 17 00:42:13.656498 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:13.656468    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:42:13.657470 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:13.657448    2573 generic.go:356] "Generic (PLEG): container finished" podID="0b888088-01c1-4c7e-971d-320ce004b459" containerID="169717692ab9577892f7c86f0bb538bf5c93b6af469ba1c25cfe2ebfa59a8272" exitCode=7
Jul 17 00:42:13.657551 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:13.657533    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" event={"ID":"0b888088-01c1-4c7e-971d-320ce004b459","Type":"ContainerDied","Data":"169717692ab9577892f7c86f0bb538bf5c93b6af469ba1c25cfe2ebfa59a8272"}
Jul 17 00:42:13.657587 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:13.657574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" event={"ID":"0b888088-01c1-4c7e-971d-320ce004b459","Type":"ContainerStarted","Data":"86e2ec7f3491aebcc1b988c494850551a2628ede56e9a385755600135058c101"}
Jul 17 00:42:13.657653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:13.657641    2573 scope.go:117] "RemoveContainer" containerID="169717692ab9577892f7c86f0bb538bf5c93b6af469ba1c25cfe2ebfa59a8272"
Jul 17 00:42:14.664356 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:14.664322    2573 generic.go:356] "Generic (PLEG): container finished" podID="0b888088-01c1-4c7e-971d-320ce004b459" containerID="84c9dbc085697d594d6afb2cdc6ebee9a059fcd865c2ff271e22392012cc7690" exitCode=7
Jul 17 00:42:14.664775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:14.664406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" event={"ID":"0b888088-01c1-4c7e-971d-320ce004b459","Type":"ContainerDied","Data":"84c9dbc085697d594d6afb2cdc6ebee9a059fcd865c2ff271e22392012cc7690"}
Jul 17 00:42:14.664775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:14.664470    2573 scope.go:117] "RemoveContainer" containerID="169717692ab9577892f7c86f0bb538bf5c93b6af469ba1c25cfe2ebfa59a8272"
Jul 17 00:42:14.664775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:14.664681    2573 scope.go:117] "RemoveContainer" containerID="84c9dbc085697d594d6afb2cdc6ebee9a059fcd865c2ff271e22392012cc7690"
Jul 17 00:42:14.664984 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:14.664778    2573 scope.go:117] "RemoveContainer" containerID="b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f"
Jul 17 00:42:14.664984 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:14.664880    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md_odh-ai-gateway-infra(0b888088-01c1-4c7e-971d-320ce004b459)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" podUID="0b888088-01c1-4c7e-971d-320ce004b459"
Jul 17 00:42:14.665073 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:14.664978    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:42:15.672090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:15.672061    2573 scope.go:117] "RemoveContainer" containerID="84c9dbc085697d594d6afb2cdc6ebee9a059fcd865c2ff271e22392012cc7690"
Jul 17 00:42:15.672476 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:15.672279    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md_odh-ai-gateway-infra(0b888088-01c1-4c7e-971d-320ce004b459)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" podUID="0b888088-01c1-4c7e-971d-320ce004b459"
Jul 17 00:42:18.489774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:18.489743    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:18.490182 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:18.490099    2573 scope.go:117] "RemoveContainer" containerID="b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f"
Jul 17 00:42:18.490261 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:18.490243    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:42:27.962862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:27.962834    2573 scope.go:117] "RemoveContainer" containerID="84c9dbc085697d594d6afb2cdc6ebee9a059fcd865c2ff271e22392012cc7690"
Jul 17 00:42:28.741998 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:28.741968    2573 generic.go:356] "Generic (PLEG): container finished" podID="0b888088-01c1-4c7e-971d-320ce004b459" containerID="7261c48bdd9d77567c1c893d464b29bc7d2b622cbcbbe184b76c23cea5011a80" exitCode=7
Jul 17 00:42:28.742112 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:28.742049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" event={"ID":"0b888088-01c1-4c7e-971d-320ce004b459","Type":"ContainerDied","Data":"7261c48bdd9d77567c1c893d464b29bc7d2b622cbcbbe184b76c23cea5011a80"}
Jul 17 00:42:28.742154 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:28.742115    2573 scope.go:117] "RemoveContainer" containerID="84c9dbc085697d594d6afb2cdc6ebee9a059fcd865c2ff271e22392012cc7690"
Jul 17 00:42:28.742393 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:28.742373    2573 scope.go:117] "RemoveContainer" containerID="7261c48bdd9d77567c1c893d464b29bc7d2b622cbcbbe184b76c23cea5011a80"
Jul 17 00:42:28.742584 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:28.742563    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md_odh-ai-gateway-infra(0b888088-01c1-4c7e-971d-320ce004b459)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" podUID="0b888088-01c1-4c7e-971d-320ce004b459"
Jul 17 00:42:29.767100 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:29.767063    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"]
Jul 17 00:42:29.904479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:29.904456    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"
Jul 17 00:42:30.061059 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.060984    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b888088-01c1-4c7e-971d-320ce004b459-maas-api-service-ca\") pod \"0b888088-01c1-4c7e-971d-320ce004b459\" (UID: \"0b888088-01c1-4c7e-971d-320ce004b459\") "
Jul 17 00:42:30.061376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.061351    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b888088-01c1-4c7e-971d-320ce004b459-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b888088-01c1-4c7e-971d-320ce004b459" (UID: "0b888088-01c1-4c7e-971d-320ce004b459"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:42:30.161507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.161478    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b888088-01c1-4c7e-971d-320ce004b459-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:42:30.761404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.761369    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"
Jul 17 00:42:30.761404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.761385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md" event={"ID":"0b888088-01c1-4c7e-971d-320ce004b459","Type":"ContainerDied","Data":"86e2ec7f3491aebcc1b988c494850551a2628ede56e9a385755600135058c101"}
Jul 17 00:42:30.761670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.761472    2573 scope.go:117] "RemoveContainer" containerID="7261c48bdd9d77567c1c893d464b29bc7d2b622cbcbbe184b76c23cea5011a80"
Jul 17 00:42:30.783745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.783721    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"]
Jul 17 00:42:30.788539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:30.788518    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fp7md"]
Jul 17 00:42:31.967507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:31.967479    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b888088-01c1-4c7e-971d-320ce004b459" path="/var/lib/kubelet/pods/0b888088-01c1-4c7e-971d-320ce004b459/volumes"
Jul 17 00:42:33.963137 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:33.963108    2573 scope.go:117] "RemoveContainer" containerID="b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f"
Jul 17 00:42:34.783005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:34.782973    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerID="96cc637ccac378b4e0c641bf17f463b3cf6eaf19a1883e5c094e94d9bef3207f" exitCode=1
Jul 17 00:42:34.783127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:34.783031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerDied","Data":"96cc637ccac378b4e0c641bf17f463b3cf6eaf19a1883e5c094e94d9bef3207f"}
Jul 17 00:42:34.783127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:34.783091    2573 scope.go:117] "RemoveContainer" containerID="b5ff2ecc3b54ffc8d500a88e5d2ae3a4546fcfc36564f51a263a7750b77d1e0f"
Jul 17 00:42:34.783372 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:34.783352    2573 scope.go:117] "RemoveContainer" containerID="96cc637ccac378b4e0c641bf17f463b3cf6eaf19a1883e5c094e94d9bef3207f"
Jul 17 00:42:34.783559 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:34.783536    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:42:38.490247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:38.490215    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:38.490820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:38.490269    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:42:38.490820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:38.490611    2573 scope.go:117] "RemoveContainer" containerID="96cc637ccac378b4e0c641bf17f463b3cf6eaf19a1883e5c094e94d9bef3207f"
Jul 17 00:42:38.490939 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:38.490824    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:42:49.962446 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:42:49.962411    2573 scope.go:117] "RemoveContainer" containerID="96cc637ccac378b4e0c641bf17f463b3cf6eaf19a1883e5c094e94d9bef3207f"
Jul 17 00:42:49.962851 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:42:49.962627    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:01.962180 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:01.962151    2573 scope.go:117] "RemoveContainer" containerID="96cc637ccac378b4e0c641bf17f463b3cf6eaf19a1883e5c094e94d9bef3207f"
Jul 17 00:43:02.346741 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.346706    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"]
Jul 17 00:43:02.347080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.347067    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:02.347128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.347081    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:02.347128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.347090    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:02.347128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.347096    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:02.347229 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.347157    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:02.347229 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.347166    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:02.351003 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.350985    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"
Jul 17 00:43:02.353050 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.353021    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:43:02.356332 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.356309    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"]
Jul 17 00:43:02.405256 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.405228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021197fa-4b80-4438-bfdd-4ae3d0b5cf90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm\" (UID: \"021197fa-4b80-4438-bfdd-4ae3d0b5cf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"
Jul 17 00:43:02.505750 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.505728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021197fa-4b80-4438-bfdd-4ae3d0b5cf90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm\" (UID: \"021197fa-4b80-4438-bfdd-4ae3d0b5cf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"
Jul 17 00:43:02.506347 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.506329    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021197fa-4b80-4438-bfdd-4ae3d0b5cf90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm\" (UID: \"021197fa-4b80-4438-bfdd-4ae3d0b5cf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"
Jul 17 00:43:02.663641 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.663559    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"
Jul 17 00:43:02.796471 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.796440    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"]
Jul 17 00:43:02.823669 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:43:02.823632    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021197fa_4b80_4438_bfdd_4ae3d0b5cf90.slice/crio-6e05b56a727dc1a2dea3ad26d02feed4f5e5ba14b69f84e26e3c49471759febc WatchSource:0}: Error finding container 6e05b56a727dc1a2dea3ad26d02feed4f5e5ba14b69f84e26e3c49471759febc: Status 404 returned error can't find the container with id 6e05b56a727dc1a2dea3ad26d02feed4f5e5ba14b69f84e26e3c49471759febc
Jul 17 00:43:02.928480 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.928391    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" event={"ID":"021197fa-4b80-4438-bfdd-4ae3d0b5cf90","Type":"ContainerStarted","Data":"6e05b56a727dc1a2dea3ad26d02feed4f5e5ba14b69f84e26e3c49471759febc"}
Jul 17 00:43:02.929963 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.929939    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2" exitCode=1
Jul 17 00:43:02.930130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.930009    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerDied","Data":"c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"}
Jul 17 00:43:02.930130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.930060    2573 scope.go:117] "RemoveContainer" containerID="96cc637ccac378b4e0c641bf17f463b3cf6eaf19a1883e5c094e94d9bef3207f"
Jul 17 00:43:02.930395 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:02.930369    2573 scope.go:117] "RemoveContainer" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"
Jul 17 00:43:02.930606 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:02.930582    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:03.489457 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:03.489423    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:43:03.936753 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:03.936731    2573 scope.go:117] "RemoveContainer" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"
Jul 17 00:43:03.936953 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:03.936920    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:03.938102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:03.938080    2573 generic.go:356] "Generic (PLEG): container finished" podID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerID="7ebd12563a0183a78fa948539ff8799235210e6bb2358ce1db68a8e5b1347710" exitCode=7
Jul 17 00:43:03.938192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:03.938110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" event={"ID":"021197fa-4b80-4438-bfdd-4ae3d0b5cf90","Type":"ContainerDied","Data":"7ebd12563a0183a78fa948539ff8799235210e6bb2358ce1db68a8e5b1347710"}
Jul 17 00:43:03.938293 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:03.938279    2573 scope.go:117] "RemoveContainer" containerID="7ebd12563a0183a78fa948539ff8799235210e6bb2358ce1db68a8e5b1347710"
Jul 17 00:43:04.944957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:04.944875    2573 generic.go:356] "Generic (PLEG): container finished" podID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerID="abe78fbb7bca3a13e4c8d8de6a90915f8922a4bab92faacba3e6d16bdcbd9bb6" exitCode=7
Jul 17 00:43:04.944957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:04.944932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" event={"ID":"021197fa-4b80-4438-bfdd-4ae3d0b5cf90","Type":"ContainerDied","Data":"abe78fbb7bca3a13e4c8d8de6a90915f8922a4bab92faacba3e6d16bdcbd9bb6"}
Jul 17 00:43:04.945394 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:04.944988    2573 scope.go:117] "RemoveContainer" containerID="7ebd12563a0183a78fa948539ff8799235210e6bb2358ce1db68a8e5b1347710"
Jul 17 00:43:04.945394 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:04.945117    2573 scope.go:117] "RemoveContainer" containerID="abe78fbb7bca3a13e4c8d8de6a90915f8922a4bab92faacba3e6d16bdcbd9bb6"
Jul 17 00:43:04.945394 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:04.945319    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm_odh-ai-gateway-infra(021197fa-4b80-4438-bfdd-4ae3d0b5cf90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90"
Jul 17 00:43:05.951129 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:05.951101    2573 scope.go:117] "RemoveContainer" containerID="abe78fbb7bca3a13e4c8d8de6a90915f8922a4bab92faacba3e6d16bdcbd9bb6"
Jul 17 00:43:05.951518 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:05.951249    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm_odh-ai-gateway-infra(021197fa-4b80-4438-bfdd-4ae3d0b5cf90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90"
Jul 17 00:43:08.490506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:08.490479    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:43:08.490906 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:08.490821    2573 scope.go:117] "RemoveContainer" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"
Jul 17 00:43:08.490990 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:08.490972    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:18.962913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:18.962879    2573 scope.go:117] "RemoveContainer" containerID="abe78fbb7bca3a13e4c8d8de6a90915f8922a4bab92faacba3e6d16bdcbd9bb6"
Jul 17 00:43:18.963462 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:18.963101    2573 scope.go:117] "RemoveContainer" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"
Jul 17 00:43:18.963462 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:18.963292    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:20.020564 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:20.020533    2573 generic.go:356] "Generic (PLEG): container finished" podID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerID="73eedd6adc92edf43bdf673d9df24f87aa8d800dad4327e3d0b3bd922aaa4180" exitCode=7
Jul 17 00:43:20.021090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:20.020611    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" event={"ID":"021197fa-4b80-4438-bfdd-4ae3d0b5cf90","Type":"ContainerDied","Data":"73eedd6adc92edf43bdf673d9df24f87aa8d800dad4327e3d0b3bd922aaa4180"}
Jul 17 00:43:20.021090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:20.020672    2573 scope.go:117] "RemoveContainer" containerID="abe78fbb7bca3a13e4c8d8de6a90915f8922a4bab92faacba3e6d16bdcbd9bb6"
Jul 17 00:43:20.021090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:20.020974    2573 scope.go:117] "RemoveContainer" containerID="73eedd6adc92edf43bdf673d9df24f87aa8d800dad4327e3d0b3bd922aaa4180"
Jul 17 00:43:20.021257 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:20.021197    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm_odh-ai-gateway-infra(021197fa-4b80-4438-bfdd-4ae3d0b5cf90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90"
Jul 17 00:43:21.044515 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:21.044480    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"]
Jul 17 00:43:21.181197 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:21.181172    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"
Jul 17 00:43:21.252455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:21.252416    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021197fa-4b80-4438-bfdd-4ae3d0b5cf90-maas-api-service-ca\") pod \"021197fa-4b80-4438-bfdd-4ae3d0b5cf90\" (UID: \"021197fa-4b80-4438-bfdd-4ae3d0b5cf90\") "
Jul 17 00:43:21.252741 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:21.252719    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021197fa-4b80-4438-bfdd-4ae3d0b5cf90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "021197fa-4b80-4438-bfdd-4ae3d0b5cf90" (UID: "021197fa-4b80-4438-bfdd-4ae3d0b5cf90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:43:21.353447 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:21.353384    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021197fa-4b80-4438-bfdd-4ae3d0b5cf90-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:43:22.033172 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:22.033140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm" event={"ID":"021197fa-4b80-4438-bfdd-4ae3d0b5cf90","Type":"ContainerDied","Data":"6e05b56a727dc1a2dea3ad26d02feed4f5e5ba14b69f84e26e3c49471759febc"}
Jul 17 00:43:22.033345 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:22.033196    2573 scope.go:117] "RemoveContainer" containerID="73eedd6adc92edf43bdf673d9df24f87aa8d800dad4327e3d0b3bd922aaa4180"
Jul 17 00:43:22.033345 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:22.033151    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"
Jul 17 00:43:22.048546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:22.048521    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"]
Jul 17 00:43:22.054282 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:22.054260    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-l4xtm"]
Jul 17 00:43:23.968010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:23.967980    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" path="/var/lib/kubelet/pods/021197fa-4b80-4438-bfdd-4ae3d0b5cf90/volumes"
Jul 17 00:43:32.962285 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:32.962248    2573 scope.go:117] "RemoveContainer" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"
Jul 17 00:43:32.962696 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:32.962443    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:46.963260 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:46.963185    2573 scope.go:117] "RemoveContainer" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"
Jul 17 00:43:48.169914 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:48.169872    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a" exitCode=1
Jul 17 00:43:48.170302 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:48.169946    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerDied","Data":"488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"}
Jul 17 00:43:48.170302 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:48.170008    2573 scope.go:117] "RemoveContainer" containerID="c7d7f0c22b818f4b9f58c37ec096c199b04125546132b9f8f229cb8063f5fee2"
Jul 17 00:43:48.170400 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:48.170317    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:43:48.170531 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:48.170508    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:48.489684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:48.489598    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:43:48.489684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:48.489662    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:43:49.177561 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:49.177537    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:43:49.177987 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:49.177724    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:43:52.387855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.387819    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"]
Jul 17 00:43:52.388341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388276    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:43:52.388341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388293    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:43:52.388341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388310    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:43:52.388341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388318    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:43:52.388341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388324    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:52.388341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388330    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:52.388566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388392    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:43:52.388566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388401    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b888088-01c1-4c7e-971d-320ce004b459" containerName="revoke-keys"
Jul 17 00:43:52.388566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388408    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:43:52.388566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.388417    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:43:52.392916 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.392896    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"
Jul 17 00:43:52.395492 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.395467    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:43:52.398581 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.398559    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"]
Jul 17 00:43:52.519207 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.519170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259f0a83-9a68-42a8-befd-c7f21e45cd11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb\" (UID: \"259f0a83-9a68-42a8-befd-c7f21e45cd11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"
Jul 17 00:43:52.620532 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.620500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259f0a83-9a68-42a8-befd-c7f21e45cd11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb\" (UID: \"259f0a83-9a68-42a8-befd-c7f21e45cd11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"
Jul 17 00:43:52.621201 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.621175    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259f0a83-9a68-42a8-befd-c7f21e45cd11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb\" (UID: \"259f0a83-9a68-42a8-befd-c7f21e45cd11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"
Jul 17 00:43:52.705604 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:52.705515    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"
Jul 17 00:43:53.047601 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:53.042837    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"]
Jul 17 00:43:53.199119 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:53.199084    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" event={"ID":"259f0a83-9a68-42a8-befd-c7f21e45cd11","Type":"ContainerStarted","Data":"f49dc82ec34d70648bbd9b38cd7e9d2cd8ca0dea8776473d9ede3de03818caf4"}
Jul 17 00:43:54.205403 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:54.205366    2573 generic.go:356] "Generic (PLEG): container finished" podID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerID="37d4d331db6d2eff0e88fdef3573ea10d63be126331eec56a6d381d29c12d883" exitCode=7
Jul 17 00:43:54.205840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:54.205449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" event={"ID":"259f0a83-9a68-42a8-befd-c7f21e45cd11","Type":"ContainerDied","Data":"37d4d331db6d2eff0e88fdef3573ea10d63be126331eec56a6d381d29c12d883"}
Jul 17 00:43:54.205840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:54.205610    2573 scope.go:117] "RemoveContainer" containerID="37d4d331db6d2eff0e88fdef3573ea10d63be126331eec56a6d381d29c12d883"
Jul 17 00:43:55.213080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:55.212988    2573 generic.go:356] "Generic (PLEG): container finished" podID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerID="476302b2f98c5c99b593093410f1aa182230edbea141ac0ffdcccb54ce1a9bb8" exitCode=7
Jul 17 00:43:55.213080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:55.213033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" event={"ID":"259f0a83-9a68-42a8-befd-c7f21e45cd11","Type":"ContainerDied","Data":"476302b2f98c5c99b593093410f1aa182230edbea141ac0ffdcccb54ce1a9bb8"}
Jul 17 00:43:55.213595 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:55.213093    2573 scope.go:117] "RemoveContainer" containerID="37d4d331db6d2eff0e88fdef3573ea10d63be126331eec56a6d381d29c12d883"
Jul 17 00:43:55.213595 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:55.213295    2573 scope.go:117] "RemoveContainer" containerID="476302b2f98c5c99b593093410f1aa182230edbea141ac0ffdcccb54ce1a9bb8"
Jul 17 00:43:55.213595 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:55.213529    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb_odh-ai-gateway-infra(259f0a83-9a68-42a8-befd-c7f21e45cd11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11"
Jul 17 00:43:56.220563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:43:56.220538    2573 scope.go:117] "RemoveContainer" containerID="476302b2f98c5c99b593093410f1aa182230edbea141ac0ffdcccb54ce1a9bb8"
Jul 17 00:43:56.220964 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:43:56.220761    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb_odh-ai-gateway-infra(259f0a83-9a68-42a8-befd-c7f21e45cd11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11"
Jul 17 00:44:04.962352 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:04.962322    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:44:04.962693 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:04.962470    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:44:08.962212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:08.962176    2573 scope.go:117] "RemoveContainer" containerID="476302b2f98c5c99b593093410f1aa182230edbea141ac0ffdcccb54ce1a9bb8"
Jul 17 00:44:09.289222 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:09.289193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" event={"ID":"259f0a83-9a68-42a8-befd-c7f21e45cd11","Type":"ContainerStarted","Data":"68958752686d990df94145906caa590f6b884d1eacc5bd8f4e7a31805394c9c6"}
Jul 17 00:44:09.304868 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:09.304821    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" podStartSLOduration=17.304804529 podStartE2EDuration="17.304804529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:44:09.302203391 +0000 UTC m=+1643.878355966" watchObservedRunningTime="2026-07-17 00:44:09.304804529 +0000 UTC m=+1643.880957102"
Jul 17 00:44:10.296731 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:10.296696    2573 generic.go:356] "Generic (PLEG): container finished" podID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerID="68958752686d990df94145906caa590f6b884d1eacc5bd8f4e7a31805394c9c6" exitCode=7
Jul 17 00:44:10.297195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:10.296770    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" event={"ID":"259f0a83-9a68-42a8-befd-c7f21e45cd11","Type":"ContainerDied","Data":"68958752686d990df94145906caa590f6b884d1eacc5bd8f4e7a31805394c9c6"}
Jul 17 00:44:10.297195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:10.296851    2573 scope.go:117] "RemoveContainer" containerID="476302b2f98c5c99b593093410f1aa182230edbea141ac0ffdcccb54ce1a9bb8"
Jul 17 00:44:10.297195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:10.297093    2573 scope.go:117] "RemoveContainer" containerID="68958752686d990df94145906caa590f6b884d1eacc5bd8f4e7a31805394c9c6"
Jul 17 00:44:10.297338 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:10.297298    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb_odh-ai-gateway-infra(259f0a83-9a68-42a8-befd-c7f21e45cd11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11"
Jul 17 00:44:10.311738 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:10.311712    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"]
Jul 17 00:44:11.437285 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:11.437260    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"
Jul 17 00:44:11.573593 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:11.573516    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259f0a83-9a68-42a8-befd-c7f21e45cd11-maas-api-service-ca\") pod \"259f0a83-9a68-42a8-befd-c7f21e45cd11\" (UID: \"259f0a83-9a68-42a8-befd-c7f21e45cd11\") "
Jul 17 00:44:11.573924 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:11.573885    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259f0a83-9a68-42a8-befd-c7f21e45cd11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "259f0a83-9a68-42a8-befd-c7f21e45cd11" (UID: "259f0a83-9a68-42a8-befd-c7f21e45cd11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:44:11.674175 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:11.674151    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/259f0a83-9a68-42a8-befd-c7f21e45cd11-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:44:12.310275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:12.310248    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"
Jul 17 00:44:12.310444 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:12.310286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb" event={"ID":"259f0a83-9a68-42a8-befd-c7f21e45cd11","Type":"ContainerDied","Data":"f49dc82ec34d70648bbd9b38cd7e9d2cd8ca0dea8776473d9ede3de03818caf4"}
Jul 17 00:44:12.310444 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:12.310349    2573 scope.go:117] "RemoveContainer" containerID="68958752686d990df94145906caa590f6b884d1eacc5bd8f4e7a31805394c9c6"
Jul 17 00:44:12.326429 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:12.326410    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"]
Jul 17 00:44:12.330829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:12.330810    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fgfgb"]
Jul 17 00:44:13.968487 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:13.968454    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" path="/var/lib/kubelet/pods/259f0a83-9a68-42a8-befd-c7f21e45cd11/volumes"
Jul 17 00:44:16.962089 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:16.962061    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:44:16.962462 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:16.962230    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:44:30.962204 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:30.962161    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:44:30.962833 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:30.962382    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:44:42.432416 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.432364    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"]
Jul 17 00:44:42.432993 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.432970    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:44:42.433048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.432998    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:44:42.433048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.433024    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:44:42.433048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.433033    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="021197fa-4b80-4438-bfdd-4ae3d0b5cf90" containerName="revoke-keys"
Jul 17 00:44:42.433151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.433049    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:44:42.433151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.433059    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:44:42.433210 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.433163    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:44:42.433210 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.433179    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:44:42.437028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.436983    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"
Jul 17 00:44:42.439327 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.439300    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:44:42.441184 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.441154    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"]
Jul 17 00:44:42.531531 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.531493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a4a19-6e1d-48af-9575-d75430fa65ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55\" (UID: \"4c4a4a19-6e1d-48af-9575-d75430fa65ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"
Jul 17 00:44:42.632125 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.632077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a4a19-6e1d-48af-9575-d75430fa65ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55\" (UID: \"4c4a4a19-6e1d-48af-9575-d75430fa65ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"
Jul 17 00:44:42.632731 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.632708    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a4a19-6e1d-48af-9575-d75430fa65ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55\" (UID: \"4c4a4a19-6e1d-48af-9575-d75430fa65ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"
Jul 17 00:44:42.751262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.751160    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"
Jul 17 00:44:42.892648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.892616    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"]
Jul 17 00:44:42.894903 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:44:42.894872    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4a4a19_6e1d_48af_9575_d75430fa65ec.slice/crio-f1d2fea53a40112aebaf1c27be17ab27f947b5279056f3bb8d233b6f9e362a5e WatchSource:0}: Error finding container f1d2fea53a40112aebaf1c27be17ab27f947b5279056f3bb8d233b6f9e362a5e: Status 404 returned error can't find the container with id f1d2fea53a40112aebaf1c27be17ab27f947b5279056f3bb8d233b6f9e362a5e
Jul 17 00:44:42.962398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:42.962356    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:44:42.962614 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:42.962566    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:44:43.482400 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:43.482363    2573 generic.go:356] "Generic (PLEG): container finished" podID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerID="10334ec2fadd8c2fd2761e76c6c621703853ff5f2cdbaba04757d57933d8711f" exitCode=7
Jul 17 00:44:43.482968 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:43.482406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" event={"ID":"4c4a4a19-6e1d-48af-9575-d75430fa65ec","Type":"ContainerDied","Data":"10334ec2fadd8c2fd2761e76c6c621703853ff5f2cdbaba04757d57933d8711f"}
Jul 17 00:44:43.482968 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:43.482481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" event={"ID":"4c4a4a19-6e1d-48af-9575-d75430fa65ec","Type":"ContainerStarted","Data":"f1d2fea53a40112aebaf1c27be17ab27f947b5279056f3bb8d233b6f9e362a5e"}
Jul 17 00:44:43.482968 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:43.482763    2573 scope.go:117] "RemoveContainer" containerID="10334ec2fadd8c2fd2761e76c6c621703853ff5f2cdbaba04757d57933d8711f"
Jul 17 00:44:44.491041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:44.490937    2573 generic.go:356] "Generic (PLEG): container finished" podID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerID="8203a96ff92a35a5c58353f547acc32ff6cbfd185d6b7ec7864a296d84a14ae3" exitCode=7
Jul 17 00:44:44.491041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:44.490990    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" event={"ID":"4c4a4a19-6e1d-48af-9575-d75430fa65ec","Type":"ContainerDied","Data":"8203a96ff92a35a5c58353f547acc32ff6cbfd185d6b7ec7864a296d84a14ae3"}
Jul 17 00:44:44.491575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:44.491058    2573 scope.go:117] "RemoveContainer" containerID="10334ec2fadd8c2fd2761e76c6c621703853ff5f2cdbaba04757d57933d8711f"
Jul 17 00:44:44.491575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:44.491314    2573 scope.go:117] "RemoveContainer" containerID="8203a96ff92a35a5c58353f547acc32ff6cbfd185d6b7ec7864a296d84a14ae3"
Jul 17 00:44:44.491686 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:44.491581    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55_odh-ai-gateway-infra(4c4a4a19-6e1d-48af-9575-d75430fa65ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec"
Jul 17 00:44:45.498989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:45.498963    2573 scope.go:117] "RemoveContainer" containerID="8203a96ff92a35a5c58353f547acc32ff6cbfd185d6b7ec7864a296d84a14ae3"
Jul 17 00:44:45.499400 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:45.499148    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55_odh-ai-gateway-infra(4c4a4a19-6e1d-48af-9575-d75430fa65ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec"
Jul 17 00:44:54.962613 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:54.962580    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:44:54.963070 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:54.962772    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:44:56.962815 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:56.962755    2573 scope.go:117] "RemoveContainer" containerID="8203a96ff92a35a5c58353f547acc32ff6cbfd185d6b7ec7864a296d84a14ae3"
Jul 17 00:44:57.569867 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:57.569827    2573 generic.go:356] "Generic (PLEG): container finished" podID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerID="162659a24d1e970710de87aa39dbe711fb83ec3fa58f2ab3de10559ade2c9a4d" exitCode=7
Jul 17 00:44:57.570061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:57.569899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" event={"ID":"4c4a4a19-6e1d-48af-9575-d75430fa65ec","Type":"ContainerDied","Data":"162659a24d1e970710de87aa39dbe711fb83ec3fa58f2ab3de10559ade2c9a4d"}
Jul 17 00:44:57.570061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:57.569973    2573 scope.go:117] "RemoveContainer" containerID="8203a96ff92a35a5c58353f547acc32ff6cbfd185d6b7ec7864a296d84a14ae3"
Jul 17 00:44:57.570333 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:57.570310    2573 scope.go:117] "RemoveContainer" containerID="162659a24d1e970710de87aa39dbe711fb83ec3fa58f2ab3de10559ade2c9a4d"
Jul 17 00:44:57.570559 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:44:57.570538    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55_odh-ai-gateway-infra(4c4a4a19-6e1d-48af-9575-d75430fa65ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec"
Jul 17 00:44:58.593758 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:58.593704    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"]
Jul 17 00:44:58.749906 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:58.749855    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"
Jul 17 00:44:58.769381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:58.769337    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a4a19-6e1d-48af-9575-d75430fa65ec-maas-api-service-ca\") pod \"4c4a4a19-6e1d-48af-9575-d75430fa65ec\" (UID: \"4c4a4a19-6e1d-48af-9575-d75430fa65ec\") "
Jul 17 00:44:58.769913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:58.769868    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4a4a19-6e1d-48af-9575-d75430fa65ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c4a4a19-6e1d-48af-9575-d75430fa65ec" (UID: "4c4a4a19-6e1d-48af-9575-d75430fa65ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:44:58.871146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:58.871052    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a4a19-6e1d-48af-9575-d75430fa65ec-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:44:59.584457 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:59.584418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55" event={"ID":"4c4a4a19-6e1d-48af-9575-d75430fa65ec","Type":"ContainerDied","Data":"f1d2fea53a40112aebaf1c27be17ab27f947b5279056f3bb8d233b6f9e362a5e"}
Jul 17 00:44:59.584650 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:59.584482    2573 scope.go:117] "RemoveContainer" containerID="162659a24d1e970710de87aa39dbe711fb83ec3fa58f2ab3de10559ade2c9a4d"
Jul 17 00:44:59.584650 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:59.584436    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"
Jul 17 00:44:59.608415 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:59.608379    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"]
Jul 17 00:44:59.612488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:59.612459    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-sbp55"]
Jul 17 00:44:59.968725 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:44:59.968633    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" path="/var/lib/kubelet/pods/4c4a4a19-6e1d-48af-9575-d75430fa65ec/volumes"
Jul 17 00:45:00.137590 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.137544    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d"]
Jul 17 00:45:00.138139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138117    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.138236 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138142    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.138236 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138164    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:45:00.138236 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138173    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:45:00.138236 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138206    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.138236 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138215    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.138426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138304    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.138426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138314    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.138426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.138325    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="259f0a83-9a68-42a8-befd-c7f21e45cd11" containerName="revoke-keys"
Jul 17 00:45:00.142808 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.142758    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"]
Jul 17 00:45:00.142933 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.142879    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d"
Jul 17 00:45:00.143281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.143255    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.143281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.143281    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.143393 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.143362    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4a4a19-6e1d-48af-9575-d75430fa65ec" containerName="revoke-keys"
Jul 17 00:45:00.144963 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.144933    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:45:00.147437 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.147243    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"
Jul 17 00:45:00.147931 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.147885    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d"]
Jul 17 00:45:00.151638 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.151571    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"]
Jul 17 00:45:00.157932 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.157904    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d"
Jul 17 00:45:00.164897 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.164874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"
Jul 17 00:45:00.313901 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.313560    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d"]
Jul 17 00:45:00.316092 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:45:00.316050    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7dcbf4_f9c7_46b9_8cb1_4583a5a0fae4.slice/crio-6989c2d5427bc84dffb9f9dcccb5f2dc1610b6dffb5667b66dda685ecc7b009f WatchSource:0}: Error finding container 6989c2d5427bc84dffb9f9dcccb5f2dc1610b6dffb5667b66dda685ecc7b009f: Status 404 returned error can't find the container with id 6989c2d5427bc84dffb9f9dcccb5f2dc1610b6dffb5667b66dda685ecc7b009f
Jul 17 00:45:00.545704 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.545677    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"]
Jul 17 00:45:00.548927 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:45:00.548896    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d470790_d696_4a74_9dfb_33ea4f137749.slice/crio-0749bdd2c2997ce1694ba3af206e5191292673fbeb87ef12ec7ebd526736c6eb WatchSource:0}: Error finding container 0749bdd2c2997ce1694ba3af206e5191292673fbeb87ef12ec7ebd526736c6eb: Status 404 returned error can't find the container with id 0749bdd2c2997ce1694ba3af206e5191292673fbeb87ef12ec7ebd526736c6eb
Jul 17 00:45:00.591561 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.591528    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d" event={"ID":"9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4","Type":"ContainerStarted","Data":"6989c2d5427bc84dffb9f9dcccb5f2dc1610b6dffb5667b66dda685ecc7b009f"}
Jul 17 00:45:00.593636 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:00.593609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" event={"ID":"2d470790-d696-4a74-9dfb-33ea4f137749","Type":"ContainerStarted","Data":"0749bdd2c2997ce1694ba3af206e5191292673fbeb87ef12ec7ebd526736c6eb"}
Jul 17 00:45:01.601320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:01.601283    2573 generic.go:356] "Generic (PLEG): container finished" podID="2d470790-d696-4a74-9dfb-33ea4f137749" containerID="d248c338be7084ef0b6210483cf2635b128ac6274e2fa44a55024409a0b37d81" exitCode=7
Jul 17 00:45:01.601839 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:01.601379    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" event={"ID":"2d470790-d696-4a74-9dfb-33ea4f137749","Type":"ContainerDied","Data":"d248c338be7084ef0b6210483cf2635b128ac6274e2fa44a55024409a0b37d81"}
Jul 17 00:45:01.601839 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:01.601491    2573 scope.go:117] "RemoveContainer" containerID="d248c338be7084ef0b6210483cf2635b128ac6274e2fa44a55024409a0b37d81"
Jul 17 00:45:01.603262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:01.603236    2573 generic.go:356] "Generic (PLEG): container finished" podID="9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4" containerID="f7ccc6571d35b2906c85bbfb248b32729f1fde7875b6e4c0210aab8e4779bf14" exitCode=0
Jul 17 00:45:01.603358 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:01.603316    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d" event={"ID":"9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4","Type":"ContainerDied","Data":"f7ccc6571d35b2906c85bbfb248b32729f1fde7875b6e4c0210aab8e4779bf14"}
Jul 17 00:45:02.610722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:02.610686    2573 generic.go:356] "Generic (PLEG): container finished" podID="2d470790-d696-4a74-9dfb-33ea4f137749" containerID="e9aa2317df04ab0e459bc48e3945555be996bc1c4dbdec460a2be2dcc872733c" exitCode=7
Jul 17 00:45:02.611284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:02.610777    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" event={"ID":"2d470790-d696-4a74-9dfb-33ea4f137749","Type":"ContainerDied","Data":"e9aa2317df04ab0e459bc48e3945555be996bc1c4dbdec460a2be2dcc872733c"}
Jul 17 00:45:02.611284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:02.610913    2573 scope.go:117] "RemoveContainer" containerID="d248c338be7084ef0b6210483cf2635b128ac6274e2fa44a55024409a0b37d81"
Jul 17 00:45:02.611284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:02.610956    2573 scope.go:117] "RemoveContainer" containerID="e9aa2317df04ab0e459bc48e3945555be996bc1c4dbdec460a2be2dcc872733c"
Jul 17 00:45:02.611284 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:02.611204    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl_odh-ai-gateway-infra(2d470790-d696-4a74-9dfb-33ea4f137749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" podUID="2d470790-d696-4a74-9dfb-33ea4f137749"
Jul 17 00:45:02.754811 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:02.754768    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d"
Jul 17 00:45:03.617302 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:03.617275    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d"
Jul 17 00:45:03.617725 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:03.617321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737485-gw87d" event={"ID":"9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4","Type":"ContainerDied","Data":"6989c2d5427bc84dffb9f9dcccb5f2dc1610b6dffb5667b66dda685ecc7b009f"}
Jul 17 00:45:03.617725 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:03.617370    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6989c2d5427bc84dffb9f9dcccb5f2dc1610b6dffb5667b66dda685ecc7b009f"
Jul 17 00:45:03.619293 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:03.619271    2573 scope.go:117] "RemoveContainer" containerID="e9aa2317df04ab0e459bc48e3945555be996bc1c4dbdec460a2be2dcc872733c"
Jul 17 00:45:03.619544 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:03.619520    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl_odh-ai-gateway-infra(2d470790-d696-4a74-9dfb-33ea4f137749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" podUID="2d470790-d696-4a74-9dfb-33ea4f137749"
Jul 17 00:45:05.966830 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:05.966802    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:45:05.967246 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:05.967021    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:45:15.965207 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:15.965118    2573 scope.go:117] "RemoveContainer" containerID="e9aa2317df04ab0e459bc48e3945555be996bc1c4dbdec460a2be2dcc872733c"
Jul 17 00:45:16.690026 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:16.689996    2573 generic.go:356] "Generic (PLEG): container finished" podID="2d470790-d696-4a74-9dfb-33ea4f137749" containerID="4c4cb9c146254983cf7f209b226964cf690e7d2f58f734f9472c3a5a95c38d5a" exitCode=7
Jul 17 00:45:16.690176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:16.690052    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" event={"ID":"2d470790-d696-4a74-9dfb-33ea4f137749","Type":"ContainerDied","Data":"4c4cb9c146254983cf7f209b226964cf690e7d2f58f734f9472c3a5a95c38d5a"}
Jul 17 00:45:16.690176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:16.690131    2573 scope.go:117] "RemoveContainer" containerID="e9aa2317df04ab0e459bc48e3945555be996bc1c4dbdec460a2be2dcc872733c"
Jul 17 00:45:16.690395 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:16.690377    2573 scope.go:117] "RemoveContainer" containerID="4c4cb9c146254983cf7f209b226964cf690e7d2f58f734f9472c3a5a95c38d5a"
Jul 17 00:45:16.690627 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:16.690603    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl_odh-ai-gateway-infra(2d470790-d696-4a74-9dfb-33ea4f137749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" podUID="2d470790-d696-4a74-9dfb-33ea4f137749"
Jul 17 00:45:16.962857 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:16.962748    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:45:17.696936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:17.696906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerStarted","Data":"bcb00f3f4e081056c0b0acb56e5777e720ac3c6985e0eaebc675a19c80989a3e"}
Jul 17 00:45:17.697300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:17.697127    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:45:17.718978 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:17.718952    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"]
Jul 17 00:45:17.730158 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:17.730111    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podStartSLOduration=185.661851335 podStartE2EDuration="3m10.730096468s" totalImagesPullingTime="5.068245133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:42:07 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:45:17.727918352 +0000 UTC m=+1712.304070930" watchObservedRunningTime="2026-07-17 00:45:17.730096468 +0000 UTC m=+1712.306249041"
Jul 17 00:45:17.876080 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:17.876025    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"
Jul 17 00:45:17.998968 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:17.998938    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"]
Jul 17 00:45:18.704313 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.704287    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"
Jul 17 00:45:18.704758 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.704327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl" event={"ID":"2d470790-d696-4a74-9dfb-33ea4f137749","Type":"ContainerDied","Data":"0749bdd2c2997ce1694ba3af206e5191292673fbeb87ef12ec7ebd526736c6eb"}
Jul 17 00:45:18.704758 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.704386    2573 scope.go:117] "RemoveContainer" containerID="4c4cb9c146254983cf7f209b226964cf690e7d2f58f734f9472c3a5a95c38d5a"
Jul 17 00:45:18.706601 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.706576    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerID="bcb00f3f4e081056c0b0acb56e5777e720ac3c6985e0eaebc675a19c80989a3e" exitCode=1
Jul 17 00:45:18.706711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.706615    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerDied","Data":"bcb00f3f4e081056c0b0acb56e5777e720ac3c6985e0eaebc675a19c80989a3e"}
Jul 17 00:45:18.706925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.706905    2573 scope.go:117] "RemoveContainer" containerID="bcb00f3f4e081056c0b0acb56e5777e720ac3c6985e0eaebc675a19c80989a3e"
Jul 17 00:45:18.707170 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:18.707147    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg_odh-ai-gateway-infra(5b77c73a-88ae-4de3-bb81-66f573c468b0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0"
Jul 17 00:45:18.713077 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.713063    2573 scope.go:117] "RemoveContainer" containerID="488bf914f16203e70361214a85c53a0efaf611720d8558adcbffc0a4aafb828a"
Jul 17 00:45:18.722501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.722479    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"]
Jul 17 00:45:18.724739 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:18.724721    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b7aa65f0-29737485-vsdnl"]
Jul 17 00:45:18.857555 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:18.857528    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-serving-cert: secret "maas-api-e2e-ait-b7aa65f0-serving-cert" not found
Jul 17 00:45:18.857745 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:18.857594    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls podName:5b77c73a-88ae-4de3-bb81-66f573c468b0 nodeName:}" failed. No retries permitted until 2026-07-17 00:45:19.357575821 +0000 UTC m=+1713.933728373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls") pod "maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" (UID: "5b77c73a-88ae-4de3-bb81-66f573c468b0") : secret "maas-api-e2e-ait-b7aa65f0-serving-cert" not found
Jul 17 00:45:19.363034 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:19.363005    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-serving-cert: secret "maas-api-e2e-ait-b7aa65f0-serving-cert" not found
Jul 17 00:45:19.363195 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:19.363072    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls podName:5b77c73a-88ae-4de3-bb81-66f573c468b0 nodeName:}" failed. No retries permitted until 2026-07-17 00:45:20.363052253 +0000 UTC m=+1714.939204806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls") pod "maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" (UID: "5b77c73a-88ae-4de3-bb81-66f573c468b0") : secret "maas-api-e2e-ait-b7aa65f0-serving-cert" not found
Jul 17 00:45:19.851731 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.851704    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:45:19.967922 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.967844    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" path="/var/lib/kubelet/pods/2d470790-d696-4a74-9dfb-33ea4f137749/volumes"
Jul 17 00:45:19.968103 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.968082    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b77c73a-88ae-4de3-bb81-66f573c468b0-openshift-service-ca\") pod \"5b77c73a-88ae-4de3-bb81-66f573c468b0\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") "
Jul 17 00:45:19.968159 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.968121    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzdh9\" (UniqueName: \"kubernetes.io/projected/5b77c73a-88ae-4de3-bb81-66f573c468b0-kube-api-access-zzdh9\") pod \"5b77c73a-88ae-4de3-bb81-66f573c468b0\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") "
Jul 17 00:45:19.968202 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.968183    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls\") pod \"5b77c73a-88ae-4de3-bb81-66f573c468b0\" (UID: \"5b77c73a-88ae-4de3-bb81-66f573c468b0\") "
Jul 17 00:45:19.968501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.968477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b77c73a-88ae-4de3-bb81-66f573c468b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b77c73a-88ae-4de3-bb81-66f573c468b0" (UID: "5b77c73a-88ae-4de3-bb81-66f573c468b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:45:19.970437 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.970415    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b77c73a-88ae-4de3-bb81-66f573c468b0-kube-api-access-zzdh9" (OuterVolumeSpecName: "kube-api-access-zzdh9") pod "5b77c73a-88ae-4de3-bb81-66f573c468b0" (UID: "5b77c73a-88ae-4de3-bb81-66f573c468b0"). InnerVolumeSpecName "kube-api-access-zzdh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:45:19.970555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:19.970459    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b77c73a-88ae-4de3-bb81-66f573c468b0" (UID: "5b77c73a-88ae-4de3-bb81-66f573c468b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:45:20.069381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.069352    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b77c73a-88ae-4de3-bb81-66f573c468b0-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:45:20.069381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.069376    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzdh9\" (UniqueName: \"kubernetes.io/projected/5b77c73a-88ae-4de3-bb81-66f573c468b0-kube-api-access-zzdh9\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:45:20.069381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.069385    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b77c73a-88ae-4de3-bb81-66f573c468b0-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:45:20.721131 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.721091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg" event={"ID":"5b77c73a-88ae-4de3-bb81-66f573c468b0","Type":"ContainerDied","Data":"6d69d29cddc563690e26a22f699cd5312cc0078b8ef75a92ed546c99ada36a33"}
Jul 17 00:45:20.721131 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.721110    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"
Jul 17 00:45:20.721346 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.721153    2573 scope.go:117] "RemoveContainer" containerID="bcb00f3f4e081056c0b0acb56e5777e720ac3c6985e0eaebc675a19c80989a3e"
Jul 17 00:45:20.742416 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.742388    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"]
Jul 17 00:45:20.744601 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:20.744575    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7aa65f0-7679d99b8-l4dwg"]
Jul 17 00:45:21.969621 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:21.969589    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" path="/var/lib/kubelet/pods/5b77c73a-88ae-4de3-bb81-66f573c468b0/volumes"
Jul 17 00:45:25.571064 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571031    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"]
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571471    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571487    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571499    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4" containerName="cleanup"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571505    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4" containerName="cleanup"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571512    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571518    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571526    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571531    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571538    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571543    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571549    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.571547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571554    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571562    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571567    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571575    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571580    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571629    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571637    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571644    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571649    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571654    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571659    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571666    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571671    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.572206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.571677    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d470790-d696-4a74-9dfb-33ea4f137749" containerName="cleanup"
Jul 17 00:45:25.576072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.576051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.577803 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.577767    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 00:45:25.578069 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.578050    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-28705dd7-serving-cert\""
Jul 17 00:45:25.584244 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.584218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"]
Jul 17 00:45:25.715821 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.715762    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.716066 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.715879    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ad0d9c-98c0-43df-812b-915800388d60-openshift-service-ca\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.716066 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.715986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfhv\" (UniqueName: \"kubernetes.io/projected/69ad0d9c-98c0-43df-812b-915800388d60-kube-api-access-xqfhv\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.817291 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.817260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfhv\" (UniqueName: \"kubernetes.io/projected/69ad0d9c-98c0-43df-812b-915800388d60-kube-api-access-xqfhv\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.817457 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.817321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.817457 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.817339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ad0d9c-98c0-43df-812b-915800388d60-openshift-service-ca\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.817538 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:25.817483    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-serving-cert: secret "maas-api-e2e-migrate-28705dd7-serving-cert" not found
Jul 17 00:45:25.817570 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:25.817548    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls podName:69ad0d9c-98c0-43df-812b-915800388d60 nodeName:}" failed. No retries permitted until 2026-07-17 00:45:26.317529521 +0000 UTC m=+1720.893682073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls") pod "maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" (UID: "69ad0d9c-98c0-43df-812b-915800388d60") : secret "maas-api-e2e-migrate-28705dd7-serving-cert" not found
Jul 17 00:45:25.818004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.817987    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ad0d9c-98c0-43df-812b-915800388d60-openshift-service-ca\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:25.830067 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:25.830003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfhv\" (UniqueName: \"kubernetes.io/projected/69ad0d9c-98c0-43df-812b-915800388d60-kube-api-access-xqfhv\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:26.320305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:26.320261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:26.323247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:26.323214    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls\") pod \"maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:26.488254 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:26.488220    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:26.626861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:26.626830    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"]
Jul 17 00:45:26.756934 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:26.756891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" event={"ID":"69ad0d9c-98c0-43df-812b-915800388d60","Type":"ContainerStarted","Data":"c5a96cd781450ffca024a6befb8eecee781ed7d5af6bcff86d20f90ccda1ed60"}
Jul 17 00:45:27.764341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:27.764305    2573 generic.go:356] "Generic (PLEG): container finished" podID="69ad0d9c-98c0-43df-812b-915800388d60" containerID="a5ffa7bf44081e59c9ae404c8ee7f29873f0816ff8c8f176e627f3c4821ff3f6" exitCode=1
Jul 17 00:45:27.764677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:27.764384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" event={"ID":"69ad0d9c-98c0-43df-812b-915800388d60","Type":"ContainerDied","Data":"a5ffa7bf44081e59c9ae404c8ee7f29873f0816ff8c8f176e627f3c4821ff3f6"}
Jul 17 00:45:27.764677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:27.764546    2573 scope.go:117] "RemoveContainer" containerID="a5ffa7bf44081e59c9ae404c8ee7f29873f0816ff8c8f176e627f3c4821ff3f6"
Jul 17 00:45:28.772542 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:28.772512    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" event={"ID":"69ad0d9c-98c0-43df-812b-915800388d60","Type":"ContainerStarted","Data":"73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51"}
Jul 17 00:45:28.772909 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:28.772623    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:28.787170 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:28.787123    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podStartSLOduration=2.542791269 podStartE2EDuration="3.787108784s" totalImagesPullingTime="1.244317515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:45:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:45:28.785981286 +0000 UTC m=+1723.362133862" watchObservedRunningTime="2026-07-17 00:45:28.787108784 +0000 UTC m=+1723.363261358"
Jul 17 00:45:29.779845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:29.779809    2573 generic.go:356] "Generic (PLEG): container finished" podID="69ad0d9c-98c0-43df-812b-915800388d60" containerID="73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51" exitCode=1
Jul 17 00:45:29.780239 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:29.779879    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" event={"ID":"69ad0d9c-98c0-43df-812b-915800388d60","Type":"ContainerDied","Data":"73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51"}
Jul 17 00:45:29.780239 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:29.779936    2573 scope.go:117] "RemoveContainer" containerID="a5ffa7bf44081e59c9ae404c8ee7f29873f0816ff8c8f176e627f3c4821ff3f6"
Jul 17 00:45:29.780239 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:29.780120    2573 scope.go:117] "RemoveContainer" containerID="73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51"
Jul 17 00:45:29.780362 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:29.780320    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:45:30.034496 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.034412    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"]
Jul 17 00:45:30.034993 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.034962    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:30.034993 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.034989    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:30.035171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.035005    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:30.035171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.035013    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:30.035171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.035090    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b77c73a-88ae-4de3-bb81-66f573c468b0" containerName="maas-api"
Jul 17 00:45:30.039541 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.039519    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"
Jul 17 00:45:30.042534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.042507    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:45:30.044649 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.044616    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"]
Jul 17 00:45:30.152054 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.152008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b6c96-9231-47ac-978d-4d10dae18e9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m\" (UID: \"a94b6c96-9231-47ac-978d-4d10dae18e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"
Jul 17 00:45:30.252519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.252489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b6c96-9231-47ac-978d-4d10dae18e9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m\" (UID: \"a94b6c96-9231-47ac-978d-4d10dae18e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"
Jul 17 00:45:30.253102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.253083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b6c96-9231-47ac-978d-4d10dae18e9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m\" (UID: \"a94b6c96-9231-47ac-978d-4d10dae18e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"
Jul 17 00:45:30.353325 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.353245    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"
Jul 17 00:45:30.496435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.496410    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"]
Jul 17 00:45:30.498313 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:45:30.498286    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94b6c96_9231_47ac_978d_4d10dae18e9a.slice/crio-91c32457e12ec4a7b4ba20222d36b1b318a230d2c6acbf23c5dbcd1c8d549bb2 WatchSource:0}: Error finding container 91c32457e12ec4a7b4ba20222d36b1b318a230d2c6acbf23c5dbcd1c8d549bb2: Status 404 returned error can't find the container with id 91c32457e12ec4a7b4ba20222d36b1b318a230d2c6acbf23c5dbcd1c8d549bb2
Jul 17 00:45:30.786531 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.786491    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" event={"ID":"a94b6c96-9231-47ac-978d-4d10dae18e9a","Type":"ContainerStarted","Data":"2b7a473fdb99660461f1e42cda8676ce91c9d110799c24fa9ab7b47cb9cb83fa"}
Jul 17 00:45:30.787039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.786558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" event={"ID":"a94b6c96-9231-47ac-978d-4d10dae18e9a","Type":"ContainerStarted","Data":"91c32457e12ec4a7b4ba20222d36b1b318a230d2c6acbf23c5dbcd1c8d549bb2"}
Jul 17 00:45:30.788328 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.788309    2573 scope.go:117] "RemoveContainer" containerID="73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51"
Jul 17 00:45:30.788500 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:30.788478    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:45:30.802205 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:30.802152    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" podStartSLOduration=0.802137276 podStartE2EDuration="802.137276ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:45:30.799637184 +0000 UTC m=+1725.375789757" watchObservedRunningTime="2026-07-17 00:45:30.802137276 +0000 UTC m=+1725.378289850"
Jul 17 00:45:31.795451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:31.795401    2573 generic.go:356] "Generic (PLEG): container finished" podID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerID="2b7a473fdb99660461f1e42cda8676ce91c9d110799c24fa9ab7b47cb9cb83fa" exitCode=7
Jul 17 00:45:31.795974 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:31.795471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" event={"ID":"a94b6c96-9231-47ac-978d-4d10dae18e9a","Type":"ContainerDied","Data":"2b7a473fdb99660461f1e42cda8676ce91c9d110799c24fa9ab7b47cb9cb83fa"}
Jul 17 00:45:31.795974 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:31.795658    2573 scope.go:117] "RemoveContainer" containerID="2b7a473fdb99660461f1e42cda8676ce91c9d110799c24fa9ab7b47cb9cb83fa"
Jul 17 00:45:32.465835 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.465810    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"]
Jul 17 00:45:32.469209 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.469194    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"
Jul 17 00:45:32.475453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.475431    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"]
Jul 17 00:45:32.571953 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.571929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4c954-a0be-497a-a8f6-3fbd74ab6580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9\" (UID: \"8ec4c954-a0be-497a-a8f6-3fbd74ab6580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"
Jul 17 00:45:32.673227 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.673172    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4c954-a0be-497a-a8f6-3fbd74ab6580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9\" (UID: \"8ec4c954-a0be-497a-a8f6-3fbd74ab6580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"
Jul 17 00:45:32.673744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.673718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4c954-a0be-497a-a8f6-3fbd74ab6580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9\" (UID: \"8ec4c954-a0be-497a-a8f6-3fbd74ab6580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"
Jul 17 00:45:32.781291 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.781265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"
Jul 17 00:45:32.804848 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.804816    2573 generic.go:356] "Generic (PLEG): container finished" podID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerID="d077600c57e4218935a83c9dd57a293e6267eb7ed4e61a8a482d7bce91b16369" exitCode=7
Jul 17 00:45:32.805230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.804828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" event={"ID":"a94b6c96-9231-47ac-978d-4d10dae18e9a","Type":"ContainerDied","Data":"d077600c57e4218935a83c9dd57a293e6267eb7ed4e61a8a482d7bce91b16369"}
Jul 17 00:45:32.805230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.804972    2573 scope.go:117] "RemoveContainer" containerID="2b7a473fdb99660461f1e42cda8676ce91c9d110799c24fa9ab7b47cb9cb83fa"
Jul 17 00:45:32.805338 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.805279    2573 scope.go:117] "RemoveContainer" containerID="d077600c57e4218935a83c9dd57a293e6267eb7ed4e61a8a482d7bce91b16369"
Jul 17 00:45:32.805527 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:32.805501    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m_odh-ai-gateway-infra(a94b6c96-9231-47ac-978d-4d10dae18e9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a"
Jul 17 00:45:32.912907 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:32.912876    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"]
Jul 17 00:45:32.914892 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:45:32.914867    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec4c954_a0be_497a_a8f6_3fbd74ab6580.slice/crio-599ec248ba9eecd67dcec3fb38f8761235f9d1e04cb86208216d440fd692e4fd WatchSource:0}: Error finding container 599ec248ba9eecd67dcec3fb38f8761235f9d1e04cb86208216d440fd692e4fd: Status 404 returned error can't find the container with id 599ec248ba9eecd67dcec3fb38f8761235f9d1e04cb86208216d440fd692e4fd
Jul 17 00:45:33.812120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:33.812041    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerID="081c2c0e98af15b6e0ab1d3c7935a6530b1d404d4b8457f4269f755c2c8ce22e" exitCode=6
Jul 17 00:45:33.812501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:33.812123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" event={"ID":"8ec4c954-a0be-497a-a8f6-3fbd74ab6580","Type":"ContainerDied","Data":"081c2c0e98af15b6e0ab1d3c7935a6530b1d404d4b8457f4269f755c2c8ce22e"}
Jul 17 00:45:33.812501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:33.812174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" event={"ID":"8ec4c954-a0be-497a-a8f6-3fbd74ab6580","Type":"ContainerStarted","Data":"599ec248ba9eecd67dcec3fb38f8761235f9d1e04cb86208216d440fd692e4fd"}
Jul 17 00:45:33.812501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:33.812324    2573 scope.go:117] "RemoveContainer" containerID="081c2c0e98af15b6e0ab1d3c7935a6530b1d404d4b8457f4269f755c2c8ce22e"
Jul 17 00:45:33.814055 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:33.814039    2573 scope.go:117] "RemoveContainer" containerID="d077600c57e4218935a83c9dd57a293e6267eb7ed4e61a8a482d7bce91b16369"
Jul 17 00:45:33.814208 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:33.814188    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m_odh-ai-gateway-infra(a94b6c96-9231-47ac-978d-4d10dae18e9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a"
Jul 17 00:45:34.821922 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:34.821882    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerID="c87edc784a36e445a9acc6e27d409161e9a34ceaa61af7e80420b129912c4d59" exitCode=6
Jul 17 00:45:34.821922 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:34.821919    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" event={"ID":"8ec4c954-a0be-497a-a8f6-3fbd74ab6580","Type":"ContainerDied","Data":"c87edc784a36e445a9acc6e27d409161e9a34ceaa61af7e80420b129912c4d59"}
Jul 17 00:45:34.822388 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:34.821983    2573 scope.go:117] "RemoveContainer" containerID="081c2c0e98af15b6e0ab1d3c7935a6530b1d404d4b8457f4269f755c2c8ce22e"
Jul 17 00:45:34.822388 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:34.822172    2573 scope.go:117] "RemoveContainer" containerID="c87edc784a36e445a9acc6e27d409161e9a34ceaa61af7e80420b129912c4d59"
Jul 17 00:45:34.822474 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:34.822419    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9_odh-ai-gateway-infra(8ec4c954-a0be-497a-a8f6-3fbd74ab6580)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580"
Jul 17 00:45:35.829668 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:35.829641    2573 scope.go:117] "RemoveContainer" containerID="c87edc784a36e445a9acc6e27d409161e9a34ceaa61af7e80420b129912c4d59"
Jul 17 00:45:35.830090 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:35.829879    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9_odh-ai-gateway-infra(8ec4c954-a0be-497a-a8f6-3fbd74ab6580)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580"
Jul 17 00:45:36.488608 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:36.488577    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:36.488998 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:36.488982    2573 scope.go:117] "RemoveContainer" containerID="73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51"
Jul 17 00:45:36.489181 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:36.489161    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:45:47.962476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:47.962444    2573 scope.go:117] "RemoveContainer" containerID="d077600c57e4218935a83c9dd57a293e6267eb7ed4e61a8a482d7bce91b16369"
Jul 17 00:45:48.906178 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:48.906105    2573 generic.go:356] "Generic (PLEG): container finished" podID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerID="0fac357a8912904b61cbc9f49b2ddd64f3c64a1cf94e4aa555c7ba030e3ccafa" exitCode=7
Jul 17 00:45:48.906312 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:48.906181    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" event={"ID":"a94b6c96-9231-47ac-978d-4d10dae18e9a","Type":"ContainerDied","Data":"0fac357a8912904b61cbc9f49b2ddd64f3c64a1cf94e4aa555c7ba030e3ccafa"}
Jul 17 00:45:48.906312 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:48.906243    2573 scope.go:117] "RemoveContainer" containerID="d077600c57e4218935a83c9dd57a293e6267eb7ed4e61a8a482d7bce91b16369"
Jul 17 00:45:48.906525 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:48.906495    2573 scope.go:117] "RemoveContainer" containerID="0fac357a8912904b61cbc9f49b2ddd64f3c64a1cf94e4aa555c7ba030e3ccafa"
Jul 17 00:45:48.906729 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:48.906703    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m_odh-ai-gateway-infra(a94b6c96-9231-47ac-978d-4d10dae18e9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a"
Jul 17 00:45:48.962988 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:48.962968    2573 scope.go:117] "RemoveContainer" containerID="c87edc784a36e445a9acc6e27d409161e9a34ceaa61af7e80420b129912c4d59"
Jul 17 00:45:49.912891 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:49.912820    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerID="f3205c01e3e131029367d227f4c19b8c4f513a1bfc53fc71aa10a67eb3b749db" exitCode=6
Jul 17 00:45:49.913033 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:49.912891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" event={"ID":"8ec4c954-a0be-497a-a8f6-3fbd74ab6580","Type":"ContainerDied","Data":"f3205c01e3e131029367d227f4c19b8c4f513a1bfc53fc71aa10a67eb3b749db"}
Jul 17 00:45:49.913033 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:49.912943    2573 scope.go:117] "RemoveContainer" containerID="c87edc784a36e445a9acc6e27d409161e9a34ceaa61af7e80420b129912c4d59"
Jul 17 00:45:49.913198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:49.913182    2573 scope.go:117] "RemoveContainer" containerID="f3205c01e3e131029367d227f4c19b8c4f513a1bfc53fc71aa10a67eb3b749db"
Jul 17 00:45:49.913395 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:49.913373    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9_odh-ai-gateway-infra(8ec4c954-a0be-497a-a8f6-3fbd74ab6580)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580"
Jul 17 00:45:49.932352 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:49.932332    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"]
Jul 17 00:45:49.963451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:49.963423    2573 scope.go:117] "RemoveContainer" containerID="73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51"
Jul 17 00:45:50.066513 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.066486    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"
Jul 17 00:45:50.113664 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.113634    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b6c96-9231-47ac-978d-4d10dae18e9a-maas-api-service-ca\") pod \"a94b6c96-9231-47ac-978d-4d10dae18e9a\" (UID: \"a94b6c96-9231-47ac-978d-4d10dae18e9a\") "
Jul 17 00:45:50.113994 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.113973    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94b6c96-9231-47ac-978d-4d10dae18e9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a94b6c96-9231-47ac-978d-4d10dae18e9a" (UID: "a94b6c96-9231-47ac-978d-4d10dae18e9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:45:50.214466 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.214403    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b6c96-9231-47ac-978d-4d10dae18e9a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:45:50.928394 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.928367    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"
Jul 17 00:45:50.928519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.928370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m" event={"ID":"a94b6c96-9231-47ac-978d-4d10dae18e9a","Type":"ContainerDied","Data":"91c32457e12ec4a7b4ba20222d36b1b318a230d2c6acbf23c5dbcd1c8d549bb2"}
Jul 17 00:45:50.928596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.928531    2573 scope.go:117] "RemoveContainer" containerID="0fac357a8912904b61cbc9f49b2ddd64f3c64a1cf94e4aa555c7ba030e3ccafa"
Jul 17 00:45:50.930268 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.930245    2573 generic.go:356] "Generic (PLEG): container finished" podID="69ad0d9c-98c0-43df-812b-915800388d60" containerID="205a088e79ef5c7c2fc7d21a6d8df1c2cfb8a39038e93b53d70443d5371a1797" exitCode=1
Jul 17 00:45:50.930379 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.930314    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" event={"ID":"69ad0d9c-98c0-43df-812b-915800388d60","Type":"ContainerDied","Data":"205a088e79ef5c7c2fc7d21a6d8df1c2cfb8a39038e93b53d70443d5371a1797"}
Jul 17 00:45:50.930602 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.930586    2573 scope.go:117] "RemoveContainer" containerID="205a088e79ef5c7c2fc7d21a6d8df1c2cfb8a39038e93b53d70443d5371a1797"
Jul 17 00:45:50.930755 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:50.930738    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:45:50.936575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.936485    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"]
Jul 17 00:45:50.938924 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.938910    2573 scope.go:117] "RemoveContainer" containerID="73d3b1a2e93e1be7d863eed0d94259f433897b4ccfc09f38651dffef51875f51"
Jul 17 00:45:50.974235 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.974209    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"]
Jul 17 00:45:50.977700 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:50.977667    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-99g8m"]
Jul 17 00:45:51.073650 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.073629    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"
Jul 17 00:45:51.126418 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.126356    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4c954-a0be-497a-a8f6-3fbd74ab6580-maas-api-service-ca\") pod \"8ec4c954-a0be-497a-a8f6-3fbd74ab6580\" (UID: \"8ec4c954-a0be-497a-a8f6-3fbd74ab6580\") "
Jul 17 00:45:51.126662 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.126641    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec4c954-a0be-497a-a8f6-3fbd74ab6580-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ec4c954-a0be-497a-a8f6-3fbd74ab6580" (UID: "8ec4c954-a0be-497a-a8f6-3fbd74ab6580"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:45:51.227253 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.227231    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4c954-a0be-497a-a8f6-3fbd74ab6580-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:45:51.937835 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.937810    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"
Jul 17 00:45:51.938024 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.937837    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9" event={"ID":"8ec4c954-a0be-497a-a8f6-3fbd74ab6580","Type":"ContainerDied","Data":"599ec248ba9eecd67dcec3fb38f8761235f9d1e04cb86208216d440fd692e4fd"}
Jul 17 00:45:51.938024 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.937915    2573 scope.go:117] "RemoveContainer" containerID="f3205c01e3e131029367d227f4c19b8c4f513a1bfc53fc71aa10a67eb3b749db"
Jul 17 00:45:51.958187 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.958162    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"]
Jul 17 00:45:51.961473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.961452    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-v48m9"]
Jul 17 00:45:51.967756 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.967737    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" path="/var/lib/kubelet/pods/8ec4c954-a0be-497a-a8f6-3fbd74ab6580/volumes"
Jul 17 00:45:51.968073 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:51.968059    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" path="/var/lib/kubelet/pods/a94b6c96-9231-47ac-978d-4d10dae18e9a/volumes"
Jul 17 00:45:53.772700 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:53.772668    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:53.773138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:53.773069    2573 scope.go:117] "RemoveContainer" containerID="205a088e79ef5c7c2fc7d21a6d8df1c2cfb8a39038e93b53d70443d5371a1797"
Jul 17 00:45:53.773253 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:53.773234    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:45:56.489054 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:56.488970    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:45:56.489468 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:45:56.489452    2573 scope.go:117] "RemoveContainer" containerID="205a088e79ef5c7c2fc7d21a6d8df1c2cfb8a39038e93b53d70443d5371a1797"
Jul 17 00:45:56.489684 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:45:56.489660    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:46:10.962732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:10.962700    2573 scope.go:117] "RemoveContainer" containerID="205a088e79ef5c7c2fc7d21a6d8df1c2cfb8a39038e93b53d70443d5371a1797"
Jul 17 00:46:12.044602 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:12.044577    2573 generic.go:356] "Generic (PLEG): container finished" podID="69ad0d9c-98c0-43df-812b-915800388d60" containerID="f55506f26baba083b227a40b7b428d61bd276bd3483369ab288a2a7134dd6386" exitCode=1
Jul 17 00:46:12.044997 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:12.044649    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" event={"ID":"69ad0d9c-98c0-43df-812b-915800388d60","Type":"ContainerDied","Data":"f55506f26baba083b227a40b7b428d61bd276bd3483369ab288a2a7134dd6386"}
Jul 17 00:46:12.044997 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:12.044704    2573 scope.go:117] "RemoveContainer" containerID="205a088e79ef5c7c2fc7d21a6d8df1c2cfb8a39038e93b53d70443d5371a1797"
Jul 17 00:46:12.044997 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:12.044924    2573 scope.go:117] "RemoveContainer" containerID="f55506f26baba083b227a40b7b428d61bd276bd3483369ab288a2a7134dd6386"
Jul 17 00:46:12.045167 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:12.045068    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:46:13.772843 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:13.772810    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:46:13.773221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:13.773138    2573 scope.go:117] "RemoveContainer" containerID="f55506f26baba083b227a40b7b428d61bd276bd3483369ab288a2a7134dd6386"
Jul 17 00:46:13.773323 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:13.773303    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:46:16.488851 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:16.488761    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:46:16.489212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:16.489131    2573 scope.go:117] "RemoveContainer" containerID="f55506f26baba083b227a40b7b428d61bd276bd3483369ab288a2a7134dd6386"
Jul 17 00:46:16.489329 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:16.489309    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:46:22.506774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.506739    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"]
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507109    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507120    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507128    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507134    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507148    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507153    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507160    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507164    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507174    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:22.507214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507181    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:22.507563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507251    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507259    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:22.507563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507266    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:22.507563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507272    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:22.507563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.507277    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:22.510193 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.510177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"
Jul 17 00:46:22.511978 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.511957    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:46:22.516863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.516830    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"]
Jul 17 00:46:22.582864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.582832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f819f17-7f63-422a-b30d-42b982dc2380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8\" (UID: \"4f819f17-7f63-422a-b30d-42b982dc2380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"
Jul 17 00:46:22.683944 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.683901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f819f17-7f63-422a-b30d-42b982dc2380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8\" (UID: \"4f819f17-7f63-422a-b30d-42b982dc2380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"
Jul 17 00:46:22.684568 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.684543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f819f17-7f63-422a-b30d-42b982dc2380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8\" (UID: \"4f819f17-7f63-422a-b30d-42b982dc2380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"
Jul 17 00:46:22.821947 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.821845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"
Jul 17 00:46:22.951854 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:22.951821    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"]
Jul 17 00:46:22.952174 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:46:22.952152    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f819f17_7f63_422a_b30d_42b982dc2380.slice/crio-ea6dbef6c4dbaddeb676a416d1899ba0f0f0cfc1baf286d54587459301e78559 WatchSource:0}: Error finding container ea6dbef6c4dbaddeb676a416d1899ba0f0f0cfc1baf286d54587459301e78559: Status 404 returned error can't find the container with id ea6dbef6c4dbaddeb676a416d1899ba0f0f0cfc1baf286d54587459301e78559
Jul 17 00:46:23.102896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:23.102809    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" event={"ID":"4f819f17-7f63-422a-b30d-42b982dc2380","Type":"ContainerStarted","Data":"ea6dbef6c4dbaddeb676a416d1899ba0f0f0cfc1baf286d54587459301e78559"}
Jul 17 00:46:24.109142 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:24.109105    2573 generic.go:356] "Generic (PLEG): container finished" podID="4f819f17-7f63-422a-b30d-42b982dc2380" containerID="bd2a51622cdbbc9d6d4a8127293bc0e1a14bc0fd5577ad3a901e0cc28b1b306d" exitCode=6
Jul 17 00:46:24.109544 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:24.109197    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" event={"ID":"4f819f17-7f63-422a-b30d-42b982dc2380","Type":"ContainerDied","Data":"bd2a51622cdbbc9d6d4a8127293bc0e1a14bc0fd5577ad3a901e0cc28b1b306d"}
Jul 17 00:46:24.109544 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:24.109351    2573 scope.go:117] "RemoveContainer" containerID="bd2a51622cdbbc9d6d4a8127293bc0e1a14bc0fd5577ad3a901e0cc28b1b306d"
Jul 17 00:46:25.115807 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:25.115699    2573 generic.go:356] "Generic (PLEG): container finished" podID="4f819f17-7f63-422a-b30d-42b982dc2380" containerID="54caa455a676236af5f98ad52ee79fc9aae16fd359650c0b6454301671cffa2e" exitCode=6
Jul 17 00:46:25.115807 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:25.115750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" event={"ID":"4f819f17-7f63-422a-b30d-42b982dc2380","Type":"ContainerDied","Data":"54caa455a676236af5f98ad52ee79fc9aae16fd359650c0b6454301671cffa2e"}
Jul 17 00:46:25.116297 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:25.115854    2573 scope.go:117] "RemoveContainer" containerID="bd2a51622cdbbc9d6d4a8127293bc0e1a14bc0fd5577ad3a901e0cc28b1b306d"
Jul 17 00:46:25.116297 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:25.116069    2573 scope.go:117] "RemoveContainer" containerID="54caa455a676236af5f98ad52ee79fc9aae16fd359650c0b6454301671cffa2e"
Jul 17 00:46:25.116570 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:25.116318    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8_odh-ai-gateway-infra(4f819f17-7f63-422a-b30d-42b982dc2380)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" podUID="4f819f17-7f63-422a-b30d-42b982dc2380"
Jul 17 00:46:26.122719 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:26.122691    2573 scope.go:117] "RemoveContainer" containerID="54caa455a676236af5f98ad52ee79fc9aae16fd359650c0b6454301671cffa2e"
Jul 17 00:46:26.123130 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:26.122928    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8_odh-ai-gateway-infra(4f819f17-7f63-422a-b30d-42b982dc2380)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" podUID="4f819f17-7f63-422a-b30d-42b982dc2380"
Jul 17 00:46:30.076134 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.076093    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"]
Jul 17 00:46:30.076522 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.076457    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:30.076522 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.076468    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94b6c96-9231-47ac-978d-4d10dae18e9a" containerName="revoke-keys"
Jul 17 00:46:30.076595 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.076557    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec4c954-a0be-497a-a8f6-3fbd74ab6580" containerName="revoke-keys"
Jul 17 00:46:30.080812 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.080778    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"
Jul 17 00:46:30.085417 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.085385    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"]
Jul 17 00:46:30.255152 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.255117    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53228e7f-765f-491a-a345-4ee1a20d2276-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l\" (UID: \"53228e7f-765f-491a-a345-4ee1a20d2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"
Jul 17 00:46:30.356501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.356393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53228e7f-765f-491a-a345-4ee1a20d2276-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l\" (UID: \"53228e7f-765f-491a-a345-4ee1a20d2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"
Jul 17 00:46:30.357111 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.357089    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53228e7f-765f-491a-a345-4ee1a20d2276-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l\" (UID: \"53228e7f-765f-491a-a345-4ee1a20d2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"
Jul 17 00:46:30.394689 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.394652    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"
Jul 17 00:46:30.546489 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:30.546459    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"]
Jul 17 00:46:30.549690 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:46:30.549655    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53228e7f_765f_491a_a345_4ee1a20d2276.slice/crio-87670240052a31218c16fd00483c7d8b7103b8b86b79c09cdf0de3a7852fa8ea WatchSource:0}: Error finding container 87670240052a31218c16fd00483c7d8b7103b8b86b79c09cdf0de3a7852fa8ea: Status 404 returned error can't find the container with id 87670240052a31218c16fd00483c7d8b7103b8b86b79c09cdf0de3a7852fa8ea
Jul 17 00:46:31.152816 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:31.152696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" event={"ID":"53228e7f-765f-491a-a345-4ee1a20d2276","Type":"ContainerStarted","Data":"f720d801e2122189888aef626b7760052faf5ae1994387bb4ecccd94d47f05b1"}
Jul 17 00:46:31.152816 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:31.152770    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" event={"ID":"53228e7f-765f-491a-a345-4ee1a20d2276","Type":"ContainerStarted","Data":"87670240052a31218c16fd00483c7d8b7103b8b86b79c09cdf0de3a7852fa8ea"}
Jul 17 00:46:31.169317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:31.169254    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" podStartSLOduration=1.169234769 podStartE2EDuration="1.169234769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:46:31.16720558 +0000 UTC m=+1785.743358156" watchObservedRunningTime="2026-07-17 00:46:31.169234769 +0000 UTC m=+1785.745387344"
Jul 17 00:46:31.963230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:31.963190    2573 scope.go:117] "RemoveContainer" containerID="f55506f26baba083b227a40b7b428d61bd276bd3483369ab288a2a7134dd6386"
Jul 17 00:46:31.963437 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:31.963411    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n_odh-ai-gateway-infra(69ad0d9c-98c0-43df-812b-915800388d60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" podUID="69ad0d9c-98c0-43df-812b-915800388d60"
Jul 17 00:46:32.159691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:32.159650    2573 generic.go:356] "Generic (PLEG): container finished" podID="53228e7f-765f-491a-a345-4ee1a20d2276" containerID="f720d801e2122189888aef626b7760052faf5ae1994387bb4ecccd94d47f05b1" exitCode=7
Jul 17 00:46:32.159691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:32.159695    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" event={"ID":"53228e7f-765f-491a-a345-4ee1a20d2276","Type":"ContainerDied","Data":"f720d801e2122189888aef626b7760052faf5ae1994387bb4ecccd94d47f05b1"}
Jul 17 00:46:32.160258 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:32.159975    2573 scope.go:117] "RemoveContainer" containerID="f720d801e2122189888aef626b7760052faf5ae1994387bb4ecccd94d47f05b1"
Jul 17 00:46:33.167201 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:33.167161    2573 generic.go:356] "Generic (PLEG): container finished" podID="53228e7f-765f-491a-a345-4ee1a20d2276" containerID="451190e00fa7d19f7ca05ad805bc3ad752152fbf52189e8dea0336cf357dde46" exitCode=7
Jul 17 00:46:33.167638 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:33.167225    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" event={"ID":"53228e7f-765f-491a-a345-4ee1a20d2276","Type":"ContainerDied","Data":"451190e00fa7d19f7ca05ad805bc3ad752152fbf52189e8dea0336cf357dde46"}
Jul 17 00:46:33.167638 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:33.167289    2573 scope.go:117] "RemoveContainer" containerID="f720d801e2122189888aef626b7760052faf5ae1994387bb4ecccd94d47f05b1"
Jul 17 00:46:33.167638 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:33.167488    2573 scope.go:117] "RemoveContainer" containerID="451190e00fa7d19f7ca05ad805bc3ad752152fbf52189e8dea0336cf357dde46"
Jul 17 00:46:33.167767 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:33.167721    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l_odh-ai-gateway-infra(53228e7f-765f-491a-a345-4ee1a20d2276)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" podUID="53228e7f-765f-491a-a345-4ee1a20d2276"
Jul 17 00:46:34.174972 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:34.174941    2573 scope.go:117] "RemoveContainer" containerID="451190e00fa7d19f7ca05ad805bc3ad752152fbf52189e8dea0336cf357dde46"
Jul 17 00:46:34.175385 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:34.175165    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l_odh-ai-gateway-infra(53228e7f-765f-491a-a345-4ee1a20d2276)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" podUID="53228e7f-765f-491a-a345-4ee1a20d2276"
Jul 17 00:46:35.863587 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:35.863552    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"]
Jul 17 00:46:36.011343 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.011310    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:46:36.186628 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.186545    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"
Jul 17 00:46:36.186628 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.186551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n" event={"ID":"69ad0d9c-98c0-43df-812b-915800388d60","Type":"ContainerDied","Data":"c5a96cd781450ffca024a6befb8eecee781ed7d5af6bcff86d20f90ccda1ed60"}
Jul 17 00:46:36.186860 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.186668    2573 scope.go:117] "RemoveContainer" containerID="f55506f26baba083b227a40b7b428d61bd276bd3483369ab288a2a7134dd6386"
Jul 17 00:46:36.208229 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.208200    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqfhv\" (UniqueName: \"kubernetes.io/projected/69ad0d9c-98c0-43df-812b-915800388d60-kube-api-access-xqfhv\") pod \"69ad0d9c-98c0-43df-812b-915800388d60\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") "
Jul 17 00:46:36.208421 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.208254    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ad0d9c-98c0-43df-812b-915800388d60-openshift-service-ca\") pod \"69ad0d9c-98c0-43df-812b-915800388d60\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") "
Jul 17 00:46:36.208421 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.208309    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls\") pod \"69ad0d9c-98c0-43df-812b-915800388d60\" (UID: \"69ad0d9c-98c0-43df-812b-915800388d60\") "
Jul 17 00:46:36.208671 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.208640    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ad0d9c-98c0-43df-812b-915800388d60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69ad0d9c-98c0-43df-812b-915800388d60" (UID: "69ad0d9c-98c0-43df-812b-915800388d60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:46:36.210483 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.210455    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69ad0d9c-98c0-43df-812b-915800388d60" (UID: "69ad0d9c-98c0-43df-812b-915800388d60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:46:36.210706 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.210683    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ad0d9c-98c0-43df-812b-915800388d60-kube-api-access-xqfhv" (OuterVolumeSpecName: "kube-api-access-xqfhv") pod "69ad0d9c-98c0-43df-812b-915800388d60" (UID: "69ad0d9c-98c0-43df-812b-915800388d60"). InnerVolumeSpecName "kube-api-access-xqfhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:46:36.309777 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.309734    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfhv\" (UniqueName: \"kubernetes.io/projected/69ad0d9c-98c0-43df-812b-915800388d60-kube-api-access-xqfhv\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:46:36.309777 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.309768    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ad0d9c-98c0-43df-812b-915800388d60-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:46:36.309777 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.309779    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ad0d9c-98c0-43df-812b-915800388d60-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:46:36.508745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.508700    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"]
Jul 17 00:46:36.512647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:36.512618    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-28705dd7-6775fbd6cb-qxj5n"]
Jul 17 00:46:37.969455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:37.969368    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ad0d9c-98c0-43df-812b-915800388d60" path="/var/lib/kubelet/pods/69ad0d9c-98c0-43df-812b-915800388d60/volumes"
Jul 17 00:46:39.963052 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:39.963022    2573 scope.go:117] "RemoveContainer" containerID="54caa455a676236af5f98ad52ee79fc9aae16fd359650c0b6454301671cffa2e"
Jul 17 00:46:41.218169 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:41.218133    2573 generic.go:356] "Generic (PLEG): container finished" podID="4f819f17-7f63-422a-b30d-42b982dc2380" containerID="51bfa23c82756071cb04085b3acf49eaf979cd346bc403a94dc7fd784de61720" exitCode=6
Jul 17 00:46:41.218677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:41.218212    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" event={"ID":"4f819f17-7f63-422a-b30d-42b982dc2380","Type":"ContainerDied","Data":"51bfa23c82756071cb04085b3acf49eaf979cd346bc403a94dc7fd784de61720"}
Jul 17 00:46:41.218677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:41.218291    2573 scope.go:117] "RemoveContainer" containerID="54caa455a676236af5f98ad52ee79fc9aae16fd359650c0b6454301671cffa2e"
Jul 17 00:46:41.218677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:41.218556    2573 scope.go:117] "RemoveContainer" containerID="51bfa23c82756071cb04085b3acf49eaf979cd346bc403a94dc7fd784de61720"
Jul 17 00:46:41.218841 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:41.218779    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8_odh-ai-gateway-infra(4f819f17-7f63-422a-b30d-42b982dc2380)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" podUID="4f819f17-7f63-422a-b30d-42b982dc2380"
Jul 17 00:46:42.242886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:42.242763    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"]
Jul 17 00:46:42.389688 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:42.389652    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"
Jul 17 00:46:42.464238 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:42.464193    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f819f17-7f63-422a-b30d-42b982dc2380-maas-api-service-ca\") pod \"4f819f17-7f63-422a-b30d-42b982dc2380\" (UID: \"4f819f17-7f63-422a-b30d-42b982dc2380\") "
Jul 17 00:46:42.464634 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:42.464613    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f819f17-7f63-422a-b30d-42b982dc2380-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f819f17-7f63-422a-b30d-42b982dc2380" (UID: "4f819f17-7f63-422a-b30d-42b982dc2380"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:46:42.565440 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:42.565394    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f819f17-7f63-422a-b30d-42b982dc2380-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:46:43.138007 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.137963    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"]
Jul 17 00:46:43.138570 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138547    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.138658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138573    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.138658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138587    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:43.138658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138597    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:43.138658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138617    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.138658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138625    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.138658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138636    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.138658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138646    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.139031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138677    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:43.139031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138685    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:43.139031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138774    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:43.139031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138785    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:43.139031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138813    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.139031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138821    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.139031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.138830    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:43.142531 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.142504    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.144751 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.144721    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4692a03f-serving-cert\""
Jul 17 00:46:43.144896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.144764    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 00:46:43.150457 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.150423    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"]
Jul 17 00:46:43.170981 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.170943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.171185 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.170994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfkz\" (UniqueName: \"kubernetes.io/projected/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-kube-api-access-2dfkz\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.171185 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.171075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-openshift-service-ca\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.243500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.243458    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8" event={"ID":"4f819f17-7f63-422a-b30d-42b982dc2380","Type":"ContainerDied","Data":"ea6dbef6c4dbaddeb676a416d1899ba0f0f0cfc1baf286d54587459301e78559"}
Jul 17 00:46:43.244029 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.243543    2573 scope.go:117] "RemoveContainer" containerID="51bfa23c82756071cb04085b3acf49eaf979cd346bc403a94dc7fd784de61720"
Jul 17 00:46:43.244029 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.243484    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"
Jul 17 00:46:43.270933 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.270889    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"]
Jul 17 00:46:43.272133 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.272093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-openshift-service-ca\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.272306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.272201    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.272306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.272244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfkz\" (UniqueName: \"kubernetes.io/projected/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-kube-api-access-2dfkz\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.272433 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:43.272302    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-serving-cert: secret "maas-api-e2e-ait-4692a03f-serving-cert" not found
Jul 17 00:46:43.272433 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:43.272395    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls podName:62a8b6e4-50ba-4691-933d-5d1c4f8d2003 nodeName:}" failed. No retries permitted until 2026-07-17 00:46:43.772369697 +0000 UTC m=+1798.348522256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls") pod "maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" (UID: "62a8b6e4-50ba-4691-933d-5d1c4f8d2003") : secret "maas-api-e2e-ait-4692a03f-serving-cert" not found
Jul 17 00:46:43.272904 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.272874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-openshift-service-ca\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.273761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.273740    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-kbjv8"]
Jul 17 00:46:43.280897 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.280875    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfkz\" (UniqueName: \"kubernetes.io/projected/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-kube-api-access-2dfkz\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.777429 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.777390    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.780021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.779991    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls\") pod \"maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:43.968457 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:43.968417    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" path="/var/lib/kubelet/pods/4f819f17-7f63-422a-b30d-42b982dc2380/volumes"
Jul 17 00:46:44.056004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:44.055907    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:44.199777 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:44.199740    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"]
Jul 17 00:46:44.201589 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:46:44.201561    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a8b6e4_50ba_4691_933d_5d1c4f8d2003.slice/crio-dc3e6d8441f2841cad0da562842b92180b46d7f3de3076accc0a6efe3e697dcb WatchSource:0}: Error finding container dc3e6d8441f2841cad0da562842b92180b46d7f3de3076accc0a6efe3e697dcb: Status 404 returned error can't find the container with id dc3e6d8441f2841cad0da562842b92180b46d7f3de3076accc0a6efe3e697dcb
Jul 17 00:46:44.250772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:44.250718    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerStarted","Data":"dc3e6d8441f2841cad0da562842b92180b46d7f3de3076accc0a6efe3e697dcb"}
Jul 17 00:46:44.962996 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:44.962905    2573 scope.go:117] "RemoveContainer" containerID="451190e00fa7d19f7ca05ad805bc3ad752152fbf52189e8dea0336cf357dde46"
Jul 17 00:46:45.260244 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:45.260210    2573 generic.go:356] "Generic (PLEG): container finished" podID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerID="61664e7857a35ef3beddba2a8f0c58d629e3428996aab42008295275604c47a6" exitCode=1
Jul 17 00:46:45.260743 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:45.260294    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"61664e7857a35ef3beddba2a8f0c58d629e3428996aab42008295275604c47a6"}
Jul 17 00:46:45.260743 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:45.260550    2573 scope.go:117] "RemoveContainer" containerID="61664e7857a35ef3beddba2a8f0c58d629e3428996aab42008295275604c47a6"
Jul 17 00:46:45.262404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:45.262375    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" event={"ID":"53228e7f-765f-491a-a345-4ee1a20d2276","Type":"ContainerStarted","Data":"7e4193350e955e56d3572017c89f9d6c3f1f73cd0969b55fc96ac5c094fd6739"}
Jul 17 00:46:46.269585 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.269540    2573 generic.go:356] "Generic (PLEG): container finished" podID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerID="fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5" exitCode=1
Jul 17 00:46:46.270082 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.269633    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5"}
Jul 17 00:46:46.270082 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.269710    2573 scope.go:117] "RemoveContainer" containerID="61664e7857a35ef3beddba2a8f0c58d629e3428996aab42008295275604c47a6"
Jul 17 00:46:46.270082 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.269994    2573 scope.go:117] "RemoveContainer" containerID="fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5"
Jul 17 00:46:46.270277 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:46.270192    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:46:46.271957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.271936    2573 generic.go:356] "Generic (PLEG): container finished" podID="53228e7f-765f-491a-a345-4ee1a20d2276" containerID="7e4193350e955e56d3572017c89f9d6c3f1f73cd0969b55fc96ac5c094fd6739" exitCode=6
Jul 17 00:46:46.272059 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.271968    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" event={"ID":"53228e7f-765f-491a-a345-4ee1a20d2276","Type":"ContainerDied","Data":"7e4193350e955e56d3572017c89f9d6c3f1f73cd0969b55fc96ac5c094fd6739"}
Jul 17 00:46:46.272228 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.272216    2573 scope.go:117] "RemoveContainer" containerID="7e4193350e955e56d3572017c89f9d6c3f1f73cd0969b55fc96ac5c094fd6739"
Jul 17 00:46:46.272487 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:46.272468    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l_odh-ai-gateway-infra(53228e7f-765f-491a-a345-4ee1a20d2276)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" podUID="53228e7f-765f-491a-a345-4ee1a20d2276"
Jul 17 00:46:46.281646 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.281625    2573 scope.go:117] "RemoveContainer" containerID="451190e00fa7d19f7ca05ad805bc3ad752152fbf52189e8dea0336cf357dde46"
Jul 17 00:46:46.296251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:46.296227    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"]
Jul 17 00:46:47.278892 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:47.278861    2573 scope.go:117] "RemoveContainer" containerID="fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5"
Jul 17 00:46:47.279341 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:47.279061    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:46:47.427953 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:47.427925    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"
Jul 17 00:46:47.513409 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:47.513347    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53228e7f-765f-491a-a345-4ee1a20d2276-maas-api-service-ca\") pod \"53228e7f-765f-491a-a345-4ee1a20d2276\" (UID: \"53228e7f-765f-491a-a345-4ee1a20d2276\") "
Jul 17 00:46:47.513726 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:47.513701    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53228e7f-765f-491a-a345-4ee1a20d2276-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53228e7f-765f-491a-a345-4ee1a20d2276" (UID: "53228e7f-765f-491a-a345-4ee1a20d2276"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:46:47.614411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:47.614320    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53228e7f-765f-491a-a345-4ee1a20d2276-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:46:47.794248 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:47.794144    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:46:47.814154 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:47.800101    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:46:48.271248 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271215    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"]
Jul 17 00:46:48.271654 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271633    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:46:48.271654 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271651    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271661    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271668    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271693    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271699    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271710    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271715    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271777    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271805    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f819f17-7f63-422a-b30d-42b982dc2380" containerName="revoke-keys"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271816    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ad0d9c-98c0-43df-812b-915800388d60" containerName="maas-api"
Jul 17 00:46:48.271855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.271825    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:46:48.274819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.274775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"
Jul 17 00:46:48.281647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.281606    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"]
Jul 17 00:46:48.289817 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.289758    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"
Jul 17 00:46:48.289817 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.289774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l" event={"ID":"53228e7f-765f-491a-a345-4ee1a20d2276","Type":"ContainerDied","Data":"87670240052a31218c16fd00483c7d8b7103b8b86b79c09cdf0de3a7852fa8ea"}
Jul 17 00:46:48.290070 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.289853    2573 scope.go:117] "RemoveContainer" containerID="7e4193350e955e56d3572017c89f9d6c3f1f73cd0969b55fc96ac5c094fd6739"
Jul 17 00:46:48.304882 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.304851    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"]
Jul 17 00:46:48.307306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.307281    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zlp5l"]
Jul 17 00:46:48.320020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.319983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh\" (UID: \"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"
Jul 17 00:46:48.421013 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.420974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh\" (UID: \"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"
Jul 17 00:46:48.421640 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.421618    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh\" (UID: \"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"
Jul 17 00:46:48.588418 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.588325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"
Jul 17 00:46:48.752657 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:48.752624    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"]
Jul 17 00:46:48.754361 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:46:48.754318    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3328f6f_8358_47f3_a51e_1fd7b9fc4cf0.slice/crio-60e139db7107978d305f1abfda8026954fe0b630960190de86f6a7a0c92f0524 WatchSource:0}: Error finding container 60e139db7107978d305f1abfda8026954fe0b630960190de86f6a7a0c92f0524: Status 404 returned error can't find the container with id 60e139db7107978d305f1abfda8026954fe0b630960190de86f6a7a0c92f0524
Jul 17 00:46:49.056216 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:49.056177    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:49.056620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:49.056606    2573 scope.go:117] "RemoveContainer" containerID="fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5"
Jul 17 00:46:49.056825 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:49.056805    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:46:49.296981 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:49.296936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" event={"ID":"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0","Type":"ContainerStarted","Data":"f3d30ac79ce5b1c4fc8ce6ef7bfbe613d4d23af3ab91f199b631dce15368c6e0"}
Jul 17 00:46:49.297495 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:49.297030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" event={"ID":"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0","Type":"ContainerStarted","Data":"60e139db7107978d305f1abfda8026954fe0b630960190de86f6a7a0c92f0524"}
Jul 17 00:46:49.311918 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:49.311857    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" podStartSLOduration=1.31183821 podStartE2EDuration="1.31183821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:46:49.310087387 +0000 UTC m=+1803.886239955" watchObservedRunningTime="2026-07-17 00:46:49.31183821 +0000 UTC m=+1803.887990815"
Jul 17 00:46:49.969775 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:49.969737    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" path="/var/lib/kubelet/pods/53228e7f-765f-491a-a345-4ee1a20d2276/volumes"
Jul 17 00:46:50.305846 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:50.305777    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerID="f3d30ac79ce5b1c4fc8ce6ef7bfbe613d4d23af3ab91f199b631dce15368c6e0" exitCode=7
Jul 17 00:46:50.306272 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:50.305893    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" event={"ID":"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0","Type":"ContainerDied","Data":"f3d30ac79ce5b1c4fc8ce6ef7bfbe613d4d23af3ab91f199b631dce15368c6e0"}
Jul 17 00:46:50.306272 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:50.306094    2573 scope.go:117] "RemoveContainer" containerID="f3d30ac79ce5b1c4fc8ce6ef7bfbe613d4d23af3ab91f199b631dce15368c6e0"
Jul 17 00:46:51.314808 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:51.314688    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerID="336a0c74182ec2b8139aacf08f2d906e8c56384d43bdd3e23c3a0eaac3e3cfda" exitCode=7
Jul 17 00:46:51.314808 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:51.314739    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" event={"ID":"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0","Type":"ContainerDied","Data":"336a0c74182ec2b8139aacf08f2d906e8c56384d43bdd3e23c3a0eaac3e3cfda"}
Jul 17 00:46:51.315296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:51.314820    2573 scope.go:117] "RemoveContainer" containerID="f3d30ac79ce5b1c4fc8ce6ef7bfbe613d4d23af3ab91f199b631dce15368c6e0"
Jul 17 00:46:51.315296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:51.315009    2573 scope.go:117] "RemoveContainer" containerID="336a0c74182ec2b8139aacf08f2d906e8c56384d43bdd3e23c3a0eaac3e3cfda"
Jul 17 00:46:51.315296 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:51.315255    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh_odh-ai-gateway-infra(b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0"
Jul 17 00:46:52.323202 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:52.323171    2573 scope.go:117] "RemoveContainer" containerID="336a0c74182ec2b8139aacf08f2d906e8c56384d43bdd3e23c3a0eaac3e3cfda"
Jul 17 00:46:52.323626 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:52.323388    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh_odh-ai-gateway-infra(b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0"
Jul 17 00:46:54.056978 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:54.056929    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:46:54.057383 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:46:54.057355    2573 scope.go:117] "RemoveContainer" containerID="fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5"
Jul 17 00:46:54.057560 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:46:54.057539    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:47:07.962266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:07.962236    2573 scope.go:117] "RemoveContainer" containerID="fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5"
Jul 17 00:47:07.962690 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:07.962403    2573 scope.go:117] "RemoveContainer" containerID="336a0c74182ec2b8139aacf08f2d906e8c56384d43bdd3e23c3a0eaac3e3cfda"
Jul 17 00:47:08.414757 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:08.414721    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" event={"ID":"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0","Type":"ContainerStarted","Data":"8775780b1a30300f4bbeadaf1bc3c11139c5cf921265748f1504850f628e50b6"}
Jul 17 00:47:09.421939 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.421904    2573 generic.go:356] "Generic (PLEG): container finished" podID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerID="24117950534e36119d9f31d506ecaa1fc808da81a6018c3f7480ea784907830d" exitCode=1
Jul 17 00:47:09.422414 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.421993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"24117950534e36119d9f31d506ecaa1fc808da81a6018c3f7480ea784907830d"}
Jul 17 00:47:09.422414 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.422067    2573 scope.go:117] "RemoveContainer" containerID="fb9110047275c495c9c8149d370024b6a7f9086a2f8a9ab87f2631d518fd28d5"
Jul 17 00:47:09.422553 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.422423    2573 scope.go:117] "RemoveContainer" containerID="24117950534e36119d9f31d506ecaa1fc808da81a6018c3f7480ea784907830d"
Jul 17 00:47:09.422658 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:09.422632    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:47:09.424098 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.424075    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerID="8775780b1a30300f4bbeadaf1bc3c11139c5cf921265748f1504850f628e50b6" exitCode=7
Jul 17 00:47:09.424175 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.424144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" event={"ID":"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0","Type":"ContainerDied","Data":"8775780b1a30300f4bbeadaf1bc3c11139c5cf921265748f1504850f628e50b6"}
Jul 17 00:47:09.424417 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.424404    2573 scope.go:117] "RemoveContainer" containerID="8775780b1a30300f4bbeadaf1bc3c11139c5cf921265748f1504850f628e50b6"
Jul 17 00:47:09.424581 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:09.424563    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh_odh-ai-gateway-infra(b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0"
Jul 17 00:47:09.433954 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.433934    2573 scope.go:117] "RemoveContainer" containerID="336a0c74182ec2b8139aacf08f2d906e8c56384d43bdd3e23c3a0eaac3e3cfda"
Jul 17 00:47:09.442451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:09.442424    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"]
Jul 17 00:47:10.580501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:10.580471    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"
Jul 17 00:47:10.733424 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:10.733324    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0-maas-api-service-ca\") pod \"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0\" (UID: \"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0\") "
Jul 17 00:47:10.733704 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:10.733680    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" (UID: "b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:47:10.834886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:10.834837    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:47:11.439453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:11.439412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh" event={"ID":"b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0","Type":"ContainerDied","Data":"60e139db7107978d305f1abfda8026954fe0b630960190de86f6a7a0c92f0524"}
Jul 17 00:47:11.439453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:11.439437    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"
Jul 17 00:47:11.439750 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:11.439478    2573 scope.go:117] "RemoveContainer" containerID="8775780b1a30300f4bbeadaf1bc3c11139c5cf921265748f1504850f628e50b6"
Jul 17 00:47:11.461902 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:11.461859    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"]
Jul 17 00:47:11.466910 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:11.466880    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-fnfqh"]
Jul 17 00:47:11.974950 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:11.974911    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" path="/var/lib/kubelet/pods/b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0/volumes"
Jul 17 00:47:14.057058 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:14.057019    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:47:14.057586 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:14.057086    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:47:14.057586 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:14.057469    2573 scope.go:117] "RemoveContainer" containerID="24117950534e36119d9f31d506ecaa1fc808da81a6018c3f7480ea784907830d"
Jul 17 00:47:14.057669 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:14.057649    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:47:20.117374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117334    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"]
Jul 17 00:47:20.117806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117725    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:20.117806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117736    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:20.117806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117764    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:20.117806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117770    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:20.117806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117776    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:47:20.117806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117781    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:47:20.118020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117875    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="53228e7f-765f-491a-a345-4ee1a20d2276" containerName="revoke-keys"
Jul 17 00:47:20.118020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117885    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:20.118020 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.117893    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:20.120993 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.120973    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"
Jul 17 00:47:20.122898 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.122870    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:47:20.128357 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.128325    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"]
Jul 17 00:47:20.222143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.222096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f433c-1e40-43e2-ae73-91ba69c21d5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n\" (UID: \"b46f433c-1e40-43e2-ae73-91ba69c21d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"
Jul 17 00:47:20.323320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.323276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f433c-1e40-43e2-ae73-91ba69c21d5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n\" (UID: \"b46f433c-1e40-43e2-ae73-91ba69c21d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"
Jul 17 00:47:20.324014 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.323995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f433c-1e40-43e2-ae73-91ba69c21d5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n\" (UID: \"b46f433c-1e40-43e2-ae73-91ba69c21d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"
Jul 17 00:47:20.433148 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.433044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"
Jul 17 00:47:20.574818 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:20.574759    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"]
Jul 17 00:47:20.576815 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:47:20.576769    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46f433c_1e40_43e2_ae73_91ba69c21d5b.slice/crio-7f17b8b8e94bf77dea37e31247b4a455ca5f8383e9a9427806b77526c4ce22c2 WatchSource:0}: Error finding container 7f17b8b8e94bf77dea37e31247b4a455ca5f8383e9a9427806b77526c4ce22c2: Status 404 returned error can't find the container with id 7f17b8b8e94bf77dea37e31247b4a455ca5f8383e9a9427806b77526c4ce22c2
Jul 17 00:47:21.500551 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:21.500455    2573 generic.go:356] "Generic (PLEG): container finished" podID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerID="48e6a31eeb27c8b598ba18f288eda9d770256e0d3a04e88f500dc59ad3a33fbf" exitCode=6
Jul 17 00:47:21.500959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:21.500539    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" event={"ID":"b46f433c-1e40-43e2-ae73-91ba69c21d5b","Type":"ContainerDied","Data":"48e6a31eeb27c8b598ba18f288eda9d770256e0d3a04e88f500dc59ad3a33fbf"}
Jul 17 00:47:21.500959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:21.500599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" event={"ID":"b46f433c-1e40-43e2-ae73-91ba69c21d5b","Type":"ContainerStarted","Data":"7f17b8b8e94bf77dea37e31247b4a455ca5f8383e9a9427806b77526c4ce22c2"}
Jul 17 00:47:21.500959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:21.500701    2573 scope.go:117] "RemoveContainer" containerID="48e6a31eeb27c8b598ba18f288eda9d770256e0d3a04e88f500dc59ad3a33fbf"
Jul 17 00:47:22.508660 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.508630    2573 generic.go:356] "Generic (PLEG): container finished" podID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerID="42d7ad11082ae922558acaa567160d6fc18a3fea76435d6426a30c95cb545608" exitCode=6
Jul 17 00:47:22.509167 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.508722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" event={"ID":"b46f433c-1e40-43e2-ae73-91ba69c21d5b","Type":"ContainerDied","Data":"42d7ad11082ae922558acaa567160d6fc18a3fea76435d6426a30c95cb545608"}
Jul 17 00:47:22.509167 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.508814    2573 scope.go:117] "RemoveContainer" containerID="48e6a31eeb27c8b598ba18f288eda9d770256e0d3a04e88f500dc59ad3a33fbf"
Jul 17 00:47:22.509167 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.508951    2573 scope.go:117] "RemoveContainer" containerID="42d7ad11082ae922558acaa567160d6fc18a3fea76435d6426a30c95cb545608"
Jul 17 00:47:22.509167 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:22.509155    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n_odh-ai-gateway-infra(b46f433c-1e40-43e2-ae73-91ba69c21d5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b"
Jul 17 00:47:22.555204 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.555171    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"]
Jul 17 00:47:22.555641 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.555623    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:22.555730 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.555645    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:22.555810 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.555739    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3328f6f-8358-47f3-a51e-1fd7b9fc4cf0" containerName="revoke-keys"
Jul 17 00:47:22.559989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.559964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"
Jul 17 00:47:22.565100 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.565068    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"]
Jul 17 00:47:22.641959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.641910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067272f2-09ad-4dc2-9c26-6dd5c070ed4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz\" (UID: \"067272f2-09ad-4dc2-9c26-6dd5c070ed4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"
Jul 17 00:47:22.742594 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.742552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067272f2-09ad-4dc2-9c26-6dd5c070ed4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz\" (UID: \"067272f2-09ad-4dc2-9c26-6dd5c070ed4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"
Jul 17 00:47:22.743248 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.743229    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067272f2-09ad-4dc2-9c26-6dd5c070ed4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz\" (UID: \"067272f2-09ad-4dc2-9c26-6dd5c070ed4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"
Jul 17 00:47:22.874012 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:22.873913    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"
Jul 17 00:47:23.011698 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:23.011670    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"]
Jul 17 00:47:23.014360 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:47:23.014330    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067272f2_09ad_4dc2_9c26_6dd5c070ed4d.slice/crio-773a14ebf8bd5e804dd607876fdae7afaac9c805ff00f4ab2b1d1bfe86307581 WatchSource:0}: Error finding container 773a14ebf8bd5e804dd607876fdae7afaac9c805ff00f4ab2b1d1bfe86307581: Status 404 returned error can't find the container with id 773a14ebf8bd5e804dd607876fdae7afaac9c805ff00f4ab2b1d1bfe86307581
Jul 17 00:47:23.523029 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:23.522998    2573 scope.go:117] "RemoveContainer" containerID="42d7ad11082ae922558acaa567160d6fc18a3fea76435d6426a30c95cb545608"
Jul 17 00:47:23.523496 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:23.523249    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n_odh-ai-gateway-infra(b46f433c-1e40-43e2-ae73-91ba69c21d5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b"
Jul 17 00:47:23.524653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:23.524623    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" event={"ID":"067272f2-09ad-4dc2-9c26-6dd5c070ed4d","Type":"ContainerStarted","Data":"b90d622de2463b09e3bca1fbac5eaac3b9482205e9f00596f00656d8af3560d9"}
Jul 17 00:47:23.524800 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:23.524713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" event={"ID":"067272f2-09ad-4dc2-9c26-6dd5c070ed4d","Type":"ContainerStarted","Data":"773a14ebf8bd5e804dd607876fdae7afaac9c805ff00f4ab2b1d1bfe86307581"}
Jul 17 00:47:23.549258 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:23.549197    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" podStartSLOduration=1.5491772419999998 podStartE2EDuration="1.549177242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:47:23.548452134 +0000 UTC m=+1838.124604718" watchObservedRunningTime="2026-07-17 00:47:23.549177242 +0000 UTC m=+1838.125329819"
Jul 17 00:47:24.531809 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:24.531759    2573 generic.go:356] "Generic (PLEG): container finished" podID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerID="b90d622de2463b09e3bca1fbac5eaac3b9482205e9f00596f00656d8af3560d9" exitCode=6
Jul 17 00:47:24.532318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:24.531848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" event={"ID":"067272f2-09ad-4dc2-9c26-6dd5c070ed4d","Type":"ContainerDied","Data":"b90d622de2463b09e3bca1fbac5eaac3b9482205e9f00596f00656d8af3560d9"}
Jul 17 00:47:24.532318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:24.532163    2573 scope.go:117] "RemoveContainer" containerID="b90d622de2463b09e3bca1fbac5eaac3b9482205e9f00596f00656d8af3560d9"
Jul 17 00:47:25.539495 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:25.539459    2573 generic.go:356] "Generic (PLEG): container finished" podID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerID="812a32e81dd3614901726f95aa01be55d75a590f1977849cdd715378111b57b7" exitCode=6
Jul 17 00:47:25.539928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:25.539521    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" event={"ID":"067272f2-09ad-4dc2-9c26-6dd5c070ed4d","Type":"ContainerDied","Data":"812a32e81dd3614901726f95aa01be55d75a590f1977849cdd715378111b57b7"}
Jul 17 00:47:25.539928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:25.539571    2573 scope.go:117] "RemoveContainer" containerID="b90d622de2463b09e3bca1fbac5eaac3b9482205e9f00596f00656d8af3560d9"
Jul 17 00:47:25.539928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:25.539754    2573 scope.go:117] "RemoveContainer" containerID="812a32e81dd3614901726f95aa01be55d75a590f1977849cdd715378111b57b7"
Jul 17 00:47:25.540121 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:25.540048    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz_odh-ai-gateway-infra(067272f2-09ad-4dc2-9c26-6dd5c070ed4d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d"
Jul 17 00:47:26.547144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:26.547118    2573 scope.go:117] "RemoveContainer" containerID="812a32e81dd3614901726f95aa01be55d75a590f1977849cdd715378111b57b7"
Jul 17 00:47:26.547549 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:26.547314    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz_odh-ai-gateway-infra(067272f2-09ad-4dc2-9c26-6dd5c070ed4d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d"
Jul 17 00:47:27.963209 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:27.963172    2573 scope.go:117] "RemoveContainer" containerID="24117950534e36119d9f31d506ecaa1fc808da81a6018c3f7480ea784907830d"
Jul 17 00:47:27.963624 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:27.963363    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:47:34.962721 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:34.962688    2573 scope.go:117] "RemoveContainer" containerID="42d7ad11082ae922558acaa567160d6fc18a3fea76435d6426a30c95cb545608"
Jul 17 00:47:35.600619 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:35.600584    2573 generic.go:356] "Generic (PLEG): container finished" podID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerID="96e29a3f61853666fd681d2cbd283601386cc4940dc3c597d4ca69a0cbd8dd44" exitCode=6
Jul 17 00:47:35.600822 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:35.600660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" event={"ID":"b46f433c-1e40-43e2-ae73-91ba69c21d5b","Type":"ContainerDied","Data":"96e29a3f61853666fd681d2cbd283601386cc4940dc3c597d4ca69a0cbd8dd44"}
Jul 17 00:47:35.600822 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:35.600728    2573 scope.go:117] "RemoveContainer" containerID="42d7ad11082ae922558acaa567160d6fc18a3fea76435d6426a30c95cb545608"
Jul 17 00:47:35.601050 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:35.601031    2573 scope.go:117] "RemoveContainer" containerID="96e29a3f61853666fd681d2cbd283601386cc4940dc3c597d4ca69a0cbd8dd44"
Jul 17 00:47:35.601279 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:35.601256    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n_odh-ai-gateway-infra(b46f433c-1e40-43e2-ae73-91ba69c21d5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b"
Jul 17 00:47:36.005910 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:36.005878    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"]
Jul 17 00:47:36.752297 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:36.752268    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"
Jul 17 00:47:36.872544 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:36.872443    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f433c-1e40-43e2-ae73-91ba69c21d5b-maas-api-service-ca\") pod \"b46f433c-1e40-43e2-ae73-91ba69c21d5b\" (UID: \"b46f433c-1e40-43e2-ae73-91ba69c21d5b\") "
Jul 17 00:47:36.872858 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:36.872834    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46f433c-1e40-43e2-ae73-91ba69c21d5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b46f433c-1e40-43e2-ae73-91ba69c21d5b" (UID: "b46f433c-1e40-43e2-ae73-91ba69c21d5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:47:36.973205 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:36.973164    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f433c-1e40-43e2-ae73-91ba69c21d5b-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:47:37.615500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:37.615468    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"
Jul 17 00:47:37.616013 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:37.615495    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n" event={"ID":"b46f433c-1e40-43e2-ae73-91ba69c21d5b","Type":"ContainerDied","Data":"7f17b8b8e94bf77dea37e31247b4a455ca5f8383e9a9427806b77526c4ce22c2"}
Jul 17 00:47:37.616013 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:37.615563    2573 scope.go:117] "RemoveContainer" containerID="96e29a3f61853666fd681d2cbd283601386cc4940dc3c597d4ca69a0cbd8dd44"
Jul 17 00:47:37.636500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:37.636471    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"]
Jul 17 00:47:37.639987 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:37.639959    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-p5b5n"]
Jul 17 00:47:37.968738 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:37.968656    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" path="/var/lib/kubelet/pods/b46f433c-1e40-43e2-ae73-91ba69c21d5b/volumes"
Jul 17 00:47:39.962985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:39.962950    2573 scope.go:117] "RemoveContainer" containerID="812a32e81dd3614901726f95aa01be55d75a590f1977849cdd715378111b57b7"
Jul 17 00:47:40.635763 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:40.635729    2573 generic.go:356] "Generic (PLEG): container finished" podID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerID="fe9da38383ec6bf5ee753f6c921466a3ee6fb388f57988af5e51b8c5e64e3e0f" exitCode=6
Jul 17 00:47:40.635967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:40.635811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" event={"ID":"067272f2-09ad-4dc2-9c26-6dd5c070ed4d","Type":"ContainerDied","Data":"fe9da38383ec6bf5ee753f6c921466a3ee6fb388f57988af5e51b8c5e64e3e0f"}
Jul 17 00:47:40.635967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:40.635879    2573 scope.go:117] "RemoveContainer" containerID="812a32e81dd3614901726f95aa01be55d75a590f1977849cdd715378111b57b7"
Jul 17 00:47:40.636263 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:40.636246    2573 scope.go:117] "RemoveContainer" containerID="fe9da38383ec6bf5ee753f6c921466a3ee6fb388f57988af5e51b8c5e64e3e0f"
Jul 17 00:47:40.636527 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:40.636495    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz_odh-ai-gateway-infra(067272f2-09ad-4dc2-9c26-6dd5c070ed4d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d"
Jul 17 00:47:41.663354 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:41.663323    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"]
Jul 17 00:47:41.804865 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:41.804838    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"
Jul 17 00:47:41.918492 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:41.918398    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067272f2-09ad-4dc2-9c26-6dd5c070ed4d-maas-api-service-ca\") pod \"067272f2-09ad-4dc2-9c26-6dd5c070ed4d\" (UID: \"067272f2-09ad-4dc2-9c26-6dd5c070ed4d\") "
Jul 17 00:47:41.918828 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:41.918781    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067272f2-09ad-4dc2-9c26-6dd5c070ed4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "067272f2-09ad-4dc2-9c26-6dd5c070ed4d" (UID: "067272f2-09ad-4dc2-9c26-6dd5c070ed4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:47:42.019937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.019900    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067272f2-09ad-4dc2-9c26-6dd5c070ed4d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:47:42.649158 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.649123    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"
Jul 17 00:47:42.649373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.649164    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz" event={"ID":"067272f2-09ad-4dc2-9c26-6dd5c070ed4d","Type":"ContainerDied","Data":"773a14ebf8bd5e804dd607876fdae7afaac9c805ff00f4ab2b1d1bfe86307581"}
Jul 17 00:47:42.649373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.649237    2573 scope.go:117] "RemoveContainer" containerID="fe9da38383ec6bf5ee753f6c921466a3ee6fb388f57988af5e51b8c5e64e3e0f"
Jul 17 00:47:42.664937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.664907    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"]
Jul 17 00:47:42.667193 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.667172    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brqvz"]
Jul 17 00:47:42.962948 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.962859    2573 scope.go:117] "RemoveContainer" containerID="24117950534e36119d9f31d506ecaa1fc808da81a6018c3f7480ea784907830d"
Jul 17 00:47:42.963210 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:42.963197    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 00:47:43.968849 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:43.968809    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" path="/var/lib/kubelet/pods/067272f2-09ad-4dc2-9c26-6dd5c070ed4d/volumes"
Jul 17 00:47:44.662859 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:44.662822    2573 generic.go:356] "Generic (PLEG): container finished" podID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226" exitCode=1
Jul 17 00:47:44.663027 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:44.662875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"}
Jul 17 00:47:44.663027 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:44.662940    2573 scope.go:117] "RemoveContainer" containerID="24117950534e36119d9f31d506ecaa1fc808da81a6018c3f7480ea784907830d"
Jul 17 00:47:44.663322 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:44.663303    2573 scope.go:117] "RemoveContainer" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"
Jul 17 00:47:44.663549 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:44.663522    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:47:48.318621 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.318582    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"]
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.318962    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.318974    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.318984    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.318990    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.318998    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319004    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319012    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319018    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319026    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319032    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319041    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319048    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319103    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319110    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319116    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="067272f2-09ad-4dc2-9c26-6dd5c070ed4d" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319124    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319130    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.319130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.319136    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46f433c-1e40-43e2-ae73-91ba69c21d5b" containerName="revoke-keys"
Jul 17 00:47:48.321995 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.321972    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"
Jul 17 00:47:48.324004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.323980    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:47:48.327883 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.327861    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"]
Jul 17 00:47:48.377860 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.377805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350e437-8b9a-4e18-badd-79388b980771-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-klc82\" (UID: \"4350e437-8b9a-4e18-badd-79388b980771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"
Jul 17 00:47:48.478990 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.478952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350e437-8b9a-4e18-badd-79388b980771-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-klc82\" (UID: \"4350e437-8b9a-4e18-badd-79388b980771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"
Jul 17 00:47:48.479667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.479633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350e437-8b9a-4e18-badd-79388b980771-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-klc82\" (UID: \"4350e437-8b9a-4e18-badd-79388b980771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"
Jul 17 00:47:48.634604 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.634518    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"
Jul 17 00:47:48.774868 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:48.774839    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"]
Jul 17 00:47:48.776016 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:47:48.775985    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4350e437_8b9a_4e18_badd_79388b980771.slice/crio-f6ed0a77bc6b2fab6e918a6df20aed6ec90fd90bea7233c61894dd312823f4a1 WatchSource:0}: Error finding container f6ed0a77bc6b2fab6e918a6df20aed6ec90fd90bea7233c61894dd312823f4a1: Status 404 returned error can't find the container with id f6ed0a77bc6b2fab6e918a6df20aed6ec90fd90bea7233c61894dd312823f4a1
Jul 17 00:47:49.056954 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:49.056923    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:47:49.057293 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:49.057279    2573 scope.go:117] "RemoveContainer" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"
Jul 17 00:47:49.057459 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:49.057442    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:47:49.694507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:49.694418    2573 generic.go:356] "Generic (PLEG): container finished" podID="4350e437-8b9a-4e18-badd-79388b980771" containerID="d203ecb22e0010523ea07a4bdd43e0a214abcde1f57133a475d3fb574ec67e5f" exitCode=7
Jul 17 00:47:49.694507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:49.694464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" event={"ID":"4350e437-8b9a-4e18-badd-79388b980771","Type":"ContainerDied","Data":"d203ecb22e0010523ea07a4bdd43e0a214abcde1f57133a475d3fb574ec67e5f"}
Jul 17 00:47:49.694967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:49.694512    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" event={"ID":"4350e437-8b9a-4e18-badd-79388b980771","Type":"ContainerStarted","Data":"f6ed0a77bc6b2fab6e918a6df20aed6ec90fd90bea7233c61894dd312823f4a1"}
Jul 17 00:47:49.694967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:49.694706    2573 scope.go:117] "RemoveContainer" containerID="d203ecb22e0010523ea07a4bdd43e0a214abcde1f57133a475d3fb574ec67e5f"
Jul 17 00:47:50.702320 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:50.702280    2573 generic.go:356] "Generic (PLEG): container finished" podID="4350e437-8b9a-4e18-badd-79388b980771" containerID="9415d5724907b3ca80e5f6c750924b6938f1d9d0554448f22c8c5c915d26c842" exitCode=7
Jul 17 00:47:50.702820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:50.702350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" event={"ID":"4350e437-8b9a-4e18-badd-79388b980771","Type":"ContainerDied","Data":"9415d5724907b3ca80e5f6c750924b6938f1d9d0554448f22c8c5c915d26c842"}
Jul 17 00:47:50.702820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:50.702411    2573 scope.go:117] "RemoveContainer" containerID="d203ecb22e0010523ea07a4bdd43e0a214abcde1f57133a475d3fb574ec67e5f"
Jul 17 00:47:50.702820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:50.702601    2573 scope.go:117] "RemoveContainer" containerID="9415d5724907b3ca80e5f6c750924b6938f1d9d0554448f22c8c5c915d26c842"
Jul 17 00:47:50.702956 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:50.702841    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-klc82_odh-ai-gateway-infra(4350e437-8b9a-4e18-badd-79388b980771)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" podUID="4350e437-8b9a-4e18-badd-79388b980771"
Jul 17 00:47:51.709158 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:51.709123    2573 scope.go:117] "RemoveContainer" containerID="9415d5724907b3ca80e5f6c750924b6938f1d9d0554448f22c8c5c915d26c842"
Jul 17 00:47:51.709545 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:51.709316    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-klc82_odh-ai-gateway-infra(4350e437-8b9a-4e18-badd-79388b980771)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" podUID="4350e437-8b9a-4e18-badd-79388b980771"
Jul 17 00:47:54.056132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:54.056089    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:47:54.056595 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:47:54.056438    2573 scope.go:117] "RemoveContainer" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"
Jul 17 00:47:54.056595 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:47:54.056584    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:48:02.962945 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:02.962904    2573 scope.go:117] "RemoveContainer" containerID="9415d5724907b3ca80e5f6c750924b6938f1d9d0554448f22c8c5c915d26c842"
Jul 17 00:48:03.775253 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:03.775220    2573 generic.go:356] "Generic (PLEG): container finished" podID="4350e437-8b9a-4e18-badd-79388b980771" containerID="6c66cdd1536a11b6b3f7744e0b325b57da95035c944018efb67ce287a98cbfe3" exitCode=7
Jul 17 00:48:03.775428 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:03.775288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" event={"ID":"4350e437-8b9a-4e18-badd-79388b980771","Type":"ContainerDied","Data":"6c66cdd1536a11b6b3f7744e0b325b57da95035c944018efb67ce287a98cbfe3"}
Jul 17 00:48:03.775428 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:03.775350    2573 scope.go:117] "RemoveContainer" containerID="9415d5724907b3ca80e5f6c750924b6938f1d9d0554448f22c8c5c915d26c842"
Jul 17 00:48:03.775638 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:03.775621    2573 scope.go:117] "RemoveContainer" containerID="6c66cdd1536a11b6b3f7744e0b325b57da95035c944018efb67ce287a98cbfe3"
Jul 17 00:48:03.775881 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:03.775853    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-klc82_odh-ai-gateway-infra(4350e437-8b9a-4e18-badd-79388b980771)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" podUID="4350e437-8b9a-4e18-badd-79388b980771"
Jul 17 00:48:04.824649 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:04.824614    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"]
Jul 17 00:48:04.958552 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:04.958523    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"
Jul 17 00:48:04.962473 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:04.962450    2573 scope.go:117] "RemoveContainer" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"
Jul 17 00:48:04.962653 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:04.962612    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:48:05.034296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.034259    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350e437-8b9a-4e18-badd-79388b980771-maas-api-service-ca\") pod \"4350e437-8b9a-4e18-badd-79388b980771\" (UID: \"4350e437-8b9a-4e18-badd-79388b980771\") "
Jul 17 00:48:05.034718 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.034605    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4350e437-8b9a-4e18-badd-79388b980771-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4350e437-8b9a-4e18-badd-79388b980771" (UID: "4350e437-8b9a-4e18-badd-79388b980771"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:48:05.135938 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.135861    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350e437-8b9a-4e18-badd-79388b980771-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:48:05.788196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.788163    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"
Jul 17 00:48:05.788196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.788169    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82" event={"ID":"4350e437-8b9a-4e18-badd-79388b980771","Type":"ContainerDied","Data":"f6ed0a77bc6b2fab6e918a6df20aed6ec90fd90bea7233c61894dd312823f4a1"}
Jul 17 00:48:05.788428 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.788225    2573 scope.go:117] "RemoveContainer" containerID="6c66cdd1536a11b6b3f7744e0b325b57da95035c944018efb67ce287a98cbfe3"
Jul 17 00:48:05.808128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.808102    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"]
Jul 17 00:48:05.812359 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.812340    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-klc82"]
Jul 17 00:48:05.968378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:05.968345    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4350e437-8b9a-4e18-badd-79388b980771" path="/var/lib/kubelet/pods/4350e437-8b9a-4e18-badd-79388b980771/volumes"
Jul 17 00:48:10.159363 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159287    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"]
Jul 17 00:48:10.159711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159636    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.159711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159647    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.159711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159657    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.159711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159662    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.159711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159671    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.159711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159676    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.159933 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159729    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.159933 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.159742    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:10.163949 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.163931    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"
Jul 17 00:48:10.166161 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.166000    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:48:10.168963 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.168939    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"]
Jul 17 00:48:10.175125 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.175101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c311d02-f0ce-453e-a80a-7b50b1e9a214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx\" (UID: \"5c311d02-f0ce-453e-a80a-7b50b1e9a214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"
Jul 17 00:48:10.276302 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.276277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c311d02-f0ce-453e-a80a-7b50b1e9a214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx\" (UID: \"5c311d02-f0ce-453e-a80a-7b50b1e9a214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"
Jul 17 00:48:10.276834 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.276817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c311d02-f0ce-453e-a80a-7b50b1e9a214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx\" (UID: \"5c311d02-f0ce-453e-a80a-7b50b1e9a214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"
Jul 17 00:48:10.476382 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.476329    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"
Jul 17 00:48:10.600783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.600743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"]
Jul 17 00:48:10.601985 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:48:10.601959    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c311d02_f0ce_453e_a80a_7b50b1e9a214.slice/crio-d624d9228ecad8afb0f9b90f4edbc7f6d9e7fbd3273d850045c7507a3e599693 WatchSource:0}: Error finding container d624d9228ecad8afb0f9b90f4edbc7f6d9e7fbd3273d850045c7507a3e599693: Status 404 returned error can't find the container with id d624d9228ecad8afb0f9b90f4edbc7f6d9e7fbd3273d850045c7507a3e599693
Jul 17 00:48:10.819084 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:10.819055    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" event={"ID":"5c311d02-f0ce-453e-a80a-7b50b1e9a214","Type":"ContainerStarted","Data":"d624d9228ecad8afb0f9b90f4edbc7f6d9e7fbd3273d850045c7507a3e599693"}
Jul 17 00:48:11.825590 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:11.825559    2573 generic.go:356] "Generic (PLEG): container finished" podID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerID="2038e948abba634939a054c70a04faaff6d7d8e3e069975d353accb3f766cc22" exitCode=6
Jul 17 00:48:11.826060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:11.825639    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" event={"ID":"5c311d02-f0ce-453e-a80a-7b50b1e9a214","Type":"ContainerDied","Data":"2038e948abba634939a054c70a04faaff6d7d8e3e069975d353accb3f766cc22"}
Jul 17 00:48:11.826060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:11.825806    2573 scope.go:117] "RemoveContainer" containerID="2038e948abba634939a054c70a04faaff6d7d8e3e069975d353accb3f766cc22"
Jul 17 00:48:12.833078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:12.833048    2573 generic.go:356] "Generic (PLEG): container finished" podID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerID="038bd1c878fd6720747f4e27d0b5750cbc73080d0b7989f64ad7f76c6975941b" exitCode=6
Jul 17 00:48:12.833476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:12.833132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" event={"ID":"5c311d02-f0ce-453e-a80a-7b50b1e9a214","Type":"ContainerDied","Data":"038bd1c878fd6720747f4e27d0b5750cbc73080d0b7989f64ad7f76c6975941b"}
Jul 17 00:48:12.833476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:12.833186    2573 scope.go:117] "RemoveContainer" containerID="2038e948abba634939a054c70a04faaff6d7d8e3e069975d353accb3f766cc22"
Jul 17 00:48:12.833476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:12.833272    2573 scope.go:117] "RemoveContainer" containerID="038bd1c878fd6720747f4e27d0b5750cbc73080d0b7989f64ad7f76c6975941b"
Jul 17 00:48:12.833476 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:12.833432    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx_odh-ai-gateway-infra(5c311d02-f0ce-453e-a80a-7b50b1e9a214)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214"
Jul 17 00:48:13.840480 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:13.840457    2573 scope.go:117] "RemoveContainer" containerID="038bd1c878fd6720747f4e27d0b5750cbc73080d0b7989f64ad7f76c6975941b"
Jul 17 00:48:13.840870 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:13.840631    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx_odh-ai-gateway-infra(5c311d02-f0ce-453e-a80a-7b50b1e9a214)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214"
Jul 17 00:48:18.962451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:18.962423    2573 scope.go:117] "RemoveContainer" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"
Jul 17 00:48:18.962828 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:18.962579    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:48:22.600381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.600345    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"]
Jul 17 00:48:22.600848 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.600739    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4350e437-8b9a-4e18-badd-79388b980771" containerName="revoke-keys"
Jul 17 00:48:22.603489 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.603474    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"
Jul 17 00:48:22.611761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.611737    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"]
Jul 17 00:48:22.674428 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.674390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de60edd-1085-4ac5-a26f-5ab06e1bd586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr\" (UID: \"8de60edd-1085-4ac5-a26f-5ab06e1bd586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"
Jul 17 00:48:22.775205 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.775178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de60edd-1085-4ac5-a26f-5ab06e1bd586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr\" (UID: \"8de60edd-1085-4ac5-a26f-5ab06e1bd586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"
Jul 17 00:48:22.775756 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.775732    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de60edd-1085-4ac5-a26f-5ab06e1bd586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr\" (UID: \"8de60edd-1085-4ac5-a26f-5ab06e1bd586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"
Jul 17 00:48:22.915301 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:22.915244    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"
Jul 17 00:48:23.044325 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:23.044298    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"]
Jul 17 00:48:23.045301 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:48:23.045275    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de60edd_1085_4ac5_a26f_5ab06e1bd586.slice/crio-026582215dac3d9b154332e0b034d0d8070a555ccb7962b47b271f808bc368b4 WatchSource:0}: Error finding container 026582215dac3d9b154332e0b034d0d8070a555ccb7962b47b271f808bc368b4: Status 404 returned error can't find the container with id 026582215dac3d9b154332e0b034d0d8070a555ccb7962b47b271f808bc368b4
Jul 17 00:48:23.893945 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:23.893892    2573 generic.go:356] "Generic (PLEG): container finished" podID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerID="bdfb09f0717acb6192486aaa6b5a6cd7454c6d1160676e03ce211922567ab30c" exitCode=6
Jul 17 00:48:23.894392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:23.893943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" event={"ID":"8de60edd-1085-4ac5-a26f-5ab06e1bd586","Type":"ContainerDied","Data":"bdfb09f0717acb6192486aaa6b5a6cd7454c6d1160676e03ce211922567ab30c"}
Jul 17 00:48:23.894392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:23.894001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" event={"ID":"8de60edd-1085-4ac5-a26f-5ab06e1bd586","Type":"ContainerStarted","Data":"026582215dac3d9b154332e0b034d0d8070a555ccb7962b47b271f808bc368b4"}
Jul 17 00:48:23.894392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:23.894171    2573 scope.go:117] "RemoveContainer" containerID="bdfb09f0717acb6192486aaa6b5a6cd7454c6d1160676e03ce211922567ab30c"
Jul 17 00:48:24.900624 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:24.900539    2573 generic.go:356] "Generic (PLEG): container finished" podID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerID="d6041b7f4f24770aec2232406a03463fbe22aad04e41cd0347a1033ef31d5f6d" exitCode=6
Jul 17 00:48:24.901018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:24.900626    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" event={"ID":"8de60edd-1085-4ac5-a26f-5ab06e1bd586","Type":"ContainerDied","Data":"d6041b7f4f24770aec2232406a03463fbe22aad04e41cd0347a1033ef31d5f6d"}
Jul 17 00:48:24.901018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:24.900703    2573 scope.go:117] "RemoveContainer" containerID="bdfb09f0717acb6192486aaa6b5a6cd7454c6d1160676e03ce211922567ab30c"
Jul 17 00:48:24.901018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:24.900911    2573 scope.go:117] "RemoveContainer" containerID="d6041b7f4f24770aec2232406a03463fbe22aad04e41cd0347a1033ef31d5f6d"
Jul 17 00:48:24.901167 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:24.901143    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr_odh-ai-gateway-infra(8de60edd-1085-4ac5-a26f-5ab06e1bd586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586"
Jul 17 00:48:25.908519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:25.908488    2573 scope.go:117] "RemoveContainer" containerID="d6041b7f4f24770aec2232406a03463fbe22aad04e41cd0347a1033ef31d5f6d"
Jul 17 00:48:25.909021 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:25.908724    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr_odh-ai-gateway-infra(8de60edd-1085-4ac5-a26f-5ab06e1bd586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586"
Jul 17 00:48:25.967360 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:25.967324    2573 scope.go:117] "RemoveContainer" containerID="038bd1c878fd6720747f4e27d0b5750cbc73080d0b7989f64ad7f76c6975941b"
Jul 17 00:48:26.916528 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:26.916434    2573 generic.go:356] "Generic (PLEG): container finished" podID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerID="da0bda8b63c1d83a9399bb6096be47008838bc51c3f98ed8f1c51a7e72f0af9e" exitCode=6
Jul 17 00:48:26.916528 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:26.916482    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" event={"ID":"5c311d02-f0ce-453e-a80a-7b50b1e9a214","Type":"ContainerDied","Data":"da0bda8b63c1d83a9399bb6096be47008838bc51c3f98ed8f1c51a7e72f0af9e"}
Jul 17 00:48:26.917063 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:26.916572    2573 scope.go:117] "RemoveContainer" containerID="038bd1c878fd6720747f4e27d0b5750cbc73080d0b7989f64ad7f76c6975941b"
Jul 17 00:48:26.917063 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:26.916928    2573 scope.go:117] "RemoveContainer" containerID="da0bda8b63c1d83a9399bb6096be47008838bc51c3f98ed8f1c51a7e72f0af9e"
Jul 17 00:48:26.917183 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:26.917159    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx_odh-ai-gateway-infra(5c311d02-f0ce-453e-a80a-7b50b1e9a214)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214"
Jul 17 00:48:27.939814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:27.939764    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"]
Jul 17 00:48:28.082979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.082936    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"
Jul 17 00:48:28.119615 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.119585    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c311d02-f0ce-453e-a80a-7b50b1e9a214-maas-api-service-ca\") pod \"5c311d02-f0ce-453e-a80a-7b50b1e9a214\" (UID: \"5c311d02-f0ce-453e-a80a-7b50b1e9a214\") "
Jul 17 00:48:28.119919 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.119894    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c311d02-f0ce-453e-a80a-7b50b1e9a214-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c311d02-f0ce-453e-a80a-7b50b1e9a214" (UID: "5c311d02-f0ce-453e-a80a-7b50b1e9a214"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:48:28.220333 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.220275    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c311d02-f0ce-453e-a80a-7b50b1e9a214-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:48:28.932151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.932128    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"
Jul 17 00:48:28.932326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.932122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx" event={"ID":"5c311d02-f0ce-453e-a80a-7b50b1e9a214","Type":"ContainerDied","Data":"d624d9228ecad8afb0f9b90f4edbc7f6d9e7fbd3273d850045c7507a3e599693"}
Jul 17 00:48:28.932326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.932265    2573 scope.go:117] "RemoveContainer" containerID="da0bda8b63c1d83a9399bb6096be47008838bc51c3f98ed8f1c51a7e72f0af9e"
Jul 17 00:48:28.956533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.956505    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"]
Jul 17 00:48:28.959089 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:28.959066    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6l8nx"]
Jul 17 00:48:29.968148 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:29.968114    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" path="/var/lib/kubelet/pods/5c311d02-f0ce-453e-a80a-7b50b1e9a214/volumes"
Jul 17 00:48:30.962561 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:30.962534    2573 scope.go:117] "RemoveContainer" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"
Jul 17 00:48:31.960272 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:31.952601    2573 generic.go:356] "Generic (PLEG): container finished" podID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a" exitCode=1
Jul 17 00:48:31.960272 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:31.952749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"}
Jul 17 00:48:31.960272 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:31.952868    2573 scope.go:117] "RemoveContainer" containerID="2e3d6902e4f357c83cf95a36f1967f40c4af9e426c15e1b65b385aa8f0d33226"
Jul 17 00:48:31.960272 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:31.953444    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:48:31.960272 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:31.953883    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:48:34.056847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:34.056773    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:48:34.057267 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:34.056871    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:48:34.057267 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:34.057215    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:48:34.057409 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:34.057389    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:48:38.359478 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.359433    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"]
Jul 17 00:48:38.360069 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.360044    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:48:38.360114 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.360073    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:48:38.360114 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.360108    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:48:38.360177 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.360117    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:48:38.360222 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.360209    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:48:38.360256 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.360228    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:48:38.360256 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.360242    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:48:38.365398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.365140    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"
Jul 17 00:48:38.369134 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.369013    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"]
Jul 17 00:48:38.511258 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.511215    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80e1eee-76b5-42c3-a3af-5569777dfaa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq\" (UID: \"e80e1eee-76b5-42c3-a3af-5569777dfaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"
Jul 17 00:48:38.612172 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.612079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80e1eee-76b5-42c3-a3af-5569777dfaa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq\" (UID: \"e80e1eee-76b5-42c3-a3af-5569777dfaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"
Jul 17 00:48:38.612842 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.612824    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80e1eee-76b5-42c3-a3af-5569777dfaa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq\" (UID: \"e80e1eee-76b5-42c3-a3af-5569777dfaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"
Jul 17 00:48:38.679270 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.679231    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"
Jul 17 00:48:38.820328 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.820261    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"]
Jul 17 00:48:38.822875 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:48:38.822842    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80e1eee_76b5_42c3_a3af_5569777dfaa6.slice/crio-693dc164a96d43e579f587e2a42387835459e208a5a196409354191bf5e87c06 WatchSource:0}: Error finding container 693dc164a96d43e579f587e2a42387835459e208a5a196409354191bf5e87c06: Status 404 returned error can't find the container with id 693dc164a96d43e579f587e2a42387835459e208a5a196409354191bf5e87c06
Jul 17 00:48:38.963015 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.962980    2573 scope.go:117] "RemoveContainer" containerID="d6041b7f4f24770aec2232406a03463fbe22aad04e41cd0347a1033ef31d5f6d"
Jul 17 00:48:38.995012 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:38.994969    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" event={"ID":"e80e1eee-76b5-42c3-a3af-5569777dfaa6","Type":"ContainerStarted","Data":"693dc164a96d43e579f587e2a42387835459e208a5a196409354191bf5e87c06"}
Jul 17 00:48:40.002038 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:40.001937    2573 generic.go:356] "Generic (PLEG): container finished" podID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerID="2f6d09c1185f170c3e710afe65a6257d377a18f09dfbcec1077f12a77aed3118" exitCode=6
Jul 17 00:48:40.002038 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:40.002010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" event={"ID":"8de60edd-1085-4ac5-a26f-5ab06e1bd586","Type":"ContainerDied","Data":"2f6d09c1185f170c3e710afe65a6257d377a18f09dfbcec1077f12a77aed3118"}
Jul 17 00:48:40.002547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:40.002069    2573 scope.go:117] "RemoveContainer" containerID="d6041b7f4f24770aec2232406a03463fbe22aad04e41cd0347a1033ef31d5f6d"
Jul 17 00:48:40.002547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:40.002415    2573 scope.go:117] "RemoveContainer" containerID="2f6d09c1185f170c3e710afe65a6257d377a18f09dfbcec1077f12a77aed3118"
Jul 17 00:48:40.002683 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:40.002660    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr_odh-ai-gateway-infra(8de60edd-1085-4ac5-a26f-5ab06e1bd586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586"
Jul 17 00:48:40.003773 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:40.003747    2573 generic.go:356] "Generic (PLEG): container finished" podID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerID="ab8e13c1366d4e2017f18eab63173b8e423dc96a3d2efed680e24892799cd5b1" exitCode=7
Jul 17 00:48:40.003903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:40.003835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" event={"ID":"e80e1eee-76b5-42c3-a3af-5569777dfaa6","Type":"ContainerDied","Data":"ab8e13c1366d4e2017f18eab63173b8e423dc96a3d2efed680e24892799cd5b1"}
Jul 17 00:48:40.003988 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:40.003976    2573 scope.go:117] "RemoveContainer" containerID="ab8e13c1366d4e2017f18eab63173b8e423dc96a3d2efed680e24892799cd5b1"
Jul 17 00:48:41.012985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.012954    2573 generic.go:356] "Generic (PLEG): container finished" podID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerID="271ba2b441970db74bb9f07f019788f51469b64a8d509a30ce40c40451e20c47" exitCode=7
Jul 17 00:48:41.013440 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.013049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" event={"ID":"e80e1eee-76b5-42c3-a3af-5569777dfaa6","Type":"ContainerDied","Data":"271ba2b441970db74bb9f07f019788f51469b64a8d509a30ce40c40451e20c47"}
Jul 17 00:48:41.013440 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.013106    2573 scope.go:117] "RemoveContainer" containerID="ab8e13c1366d4e2017f18eab63173b8e423dc96a3d2efed680e24892799cd5b1"
Jul 17 00:48:41.013440 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.013253    2573 scope.go:117] "RemoveContainer" containerID="271ba2b441970db74bb9f07f019788f51469b64a8d509a30ce40c40451e20c47"
Jul 17 00:48:41.013572 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:41.013469    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq_odh-ai-gateway-infra(e80e1eee-76b5-42c3-a3af-5569777dfaa6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6"
Jul 17 00:48:41.024604 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.024573    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"]
Jul 17 00:48:41.164474 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.164440    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"
Jul 17 00:48:41.236257 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.236222    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de60edd-1085-4ac5-a26f-5ab06e1bd586-maas-api-service-ca\") pod \"8de60edd-1085-4ac5-a26f-5ab06e1bd586\" (UID: \"8de60edd-1085-4ac5-a26f-5ab06e1bd586\") "
Jul 17 00:48:41.236642 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.236621    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de60edd-1085-4ac5-a26f-5ab06e1bd586-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8de60edd-1085-4ac5-a26f-5ab06e1bd586" (UID: "8de60edd-1085-4ac5-a26f-5ab06e1bd586"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:48:41.337927 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:41.337823    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de60edd-1085-4ac5-a26f-5ab06e1bd586-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:48:42.019928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:42.019894    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"
Jul 17 00:48:42.019928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:42.019908    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr" event={"ID":"8de60edd-1085-4ac5-a26f-5ab06e1bd586","Type":"ContainerDied","Data":"026582215dac3d9b154332e0b034d0d8070a555ccb7962b47b271f808bc368b4"}
Jul 17 00:48:42.020499 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:42.019975    2573 scope.go:117] "RemoveContainer" containerID="2f6d09c1185f170c3e710afe65a6257d377a18f09dfbcec1077f12a77aed3118"
Jul 17 00:48:42.022184 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:42.022156    2573 scope.go:117] "RemoveContainer" containerID="271ba2b441970db74bb9f07f019788f51469b64a8d509a30ce40c40451e20c47"
Jul 17 00:48:42.022614 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:42.022581    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq_odh-ai-gateway-infra(e80e1eee-76b5-42c3-a3af-5569777dfaa6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6"
Jul 17 00:48:42.049265 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:42.049229    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"]
Jul 17 00:48:42.052684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:42.052651    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-fhjbr"]
Jul 17 00:48:43.968041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:43.968009    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" path="/var/lib/kubelet/pods/8de60edd-1085-4ac5-a26f-5ab06e1bd586/volumes"
Jul 17 00:48:47.962361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:47.962325    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:48:47.962773 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:47.962519    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:48:55.966266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:55.966232    2573 scope.go:117] "RemoveContainer" containerID="271ba2b441970db74bb9f07f019788f51469b64a8d509a30ce40c40451e20c47"
Jul 17 00:48:57.103782 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:57.103747    2573 generic.go:356] "Generic (PLEG): container finished" podID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerID="250171a4439190e98afa882c9f833598c43cf8c650b0066e51a5355fdb96d261" exitCode=7
Jul 17 00:48:57.104226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:57.103825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" event={"ID":"e80e1eee-76b5-42c3-a3af-5569777dfaa6","Type":"ContainerDied","Data":"250171a4439190e98afa882c9f833598c43cf8c650b0066e51a5355fdb96d261"}
Jul 17 00:48:57.104226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:57.103890    2573 scope.go:117] "RemoveContainer" containerID="271ba2b441970db74bb9f07f019788f51469b64a8d509a30ce40c40451e20c47"
Jul 17 00:48:57.104226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:57.104207    2573 scope.go:117] "RemoveContainer" containerID="250171a4439190e98afa882c9f833598c43cf8c650b0066e51a5355fdb96d261"
Jul 17 00:48:57.104423 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:57.104398    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq_odh-ai-gateway-infra(e80e1eee-76b5-42c3-a3af-5569777dfaa6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6"
Jul 17 00:48:58.126928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:58.126890    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"]
Jul 17 00:48:58.283066 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:58.283040    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"
Jul 17 00:48:58.295914 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:58.295877    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80e1eee-76b5-42c3-a3af-5569777dfaa6-maas-api-service-ca\") pod \"e80e1eee-76b5-42c3-a3af-5569777dfaa6\" (UID: \"e80e1eee-76b5-42c3-a3af-5569777dfaa6\") "
Jul 17 00:48:58.296243 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:58.296217    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80e1eee-76b5-42c3-a3af-5569777dfaa6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e80e1eee-76b5-42c3-a3af-5569777dfaa6" (UID: "e80e1eee-76b5-42c3-a3af-5569777dfaa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:48:58.396849 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:58.396732    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80e1eee-76b5-42c3-a3af-5569777dfaa6-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:48:59.118025 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:59.117993    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"
Jul 17 00:48:59.118025 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:59.118010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq" event={"ID":"e80e1eee-76b5-42c3-a3af-5569777dfaa6","Type":"ContainerDied","Data":"693dc164a96d43e579f587e2a42387835459e208a5a196409354191bf5e87c06"}
Jul 17 00:48:59.118306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:59.118065    2573 scope.go:117] "RemoveContainer" containerID="250171a4439190e98afa882c9f833598c43cf8c650b0066e51a5355fdb96d261"
Jul 17 00:48:59.141322 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:59.141291    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"]
Jul 17 00:48:59.143590 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:59.143569    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tgmwq"]
Jul 17 00:48:59.963001 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:59.962967    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:48:59.963191 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:48:59.963156    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:48:59.968575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:48:59.968543    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" path="/var/lib/kubelet/pods/e80e1eee-76b5-42c3-a3af-5569777dfaa6/volumes"
Jul 17 00:49:00.198374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198332    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"]
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198703    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198714    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198727    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198733    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c311d02-f0ce-453e-a80a-7b50b1e9a214" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198743    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198749    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198755    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198761    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.198767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198771    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198776    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198801    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198824    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198879    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198886    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198894    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198900    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.199090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.198906    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80e1eee-76b5-42c3-a3af-5569777dfaa6" containerName="revoke-keys"
Jul 17 00:49:00.203440 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.203418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"
Jul 17 00:49:00.205691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.205663    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:49:00.208433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.208403    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"]
Jul 17 00:49:00.212411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.212045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31583a0b-885d-4356-9bc1-c409819c7404-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9\" (UID: \"31583a0b-885d-4356-9bc1-c409819c7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"
Jul 17 00:49:00.312920 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.312879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31583a0b-885d-4356-9bc1-c409819c7404-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9\" (UID: \"31583a0b-885d-4356-9bc1-c409819c7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"
Jul 17 00:49:00.313511 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.313492    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31583a0b-885d-4356-9bc1-c409819c7404-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9\" (UID: \"31583a0b-885d-4356-9bc1-c409819c7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"
Jul 17 00:49:00.516596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.516555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"
Jul 17 00:49:00.653243 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:00.653215    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"]
Jul 17 00:49:00.655248 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:49:00.655219    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31583a0b_885d_4356_9bc1_c409819c7404.slice/crio-ec62dc584fc9e6e81ef9360d56566ebcdd48da4cacc1b0167d61e5c1337b3064 WatchSource:0}: Error finding container ec62dc584fc9e6e81ef9360d56566ebcdd48da4cacc1b0167d61e5c1337b3064: Status 404 returned error can't find the container with id ec62dc584fc9e6e81ef9360d56566ebcdd48da4cacc1b0167d61e5c1337b3064
Jul 17 00:49:01.131589 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:01.131494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" event={"ID":"31583a0b-885d-4356-9bc1-c409819c7404","Type":"ContainerStarted","Data":"14f52cdf277c8efcb03d6d5f6687813fb0e2ee096e759f4de0581bba679829ed"}
Jul 17 00:49:01.131589 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:01.131552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" event={"ID":"31583a0b-885d-4356-9bc1-c409819c7404","Type":"ContainerStarted","Data":"ec62dc584fc9e6e81ef9360d56566ebcdd48da4cacc1b0167d61e5c1337b3064"}
Jul 17 00:49:01.169018 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:01.168962    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" podStartSLOduration=1.168942981 podStartE2EDuration="1.168942981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:49:01.168258008 +0000 UTC m=+1935.744410583" watchObservedRunningTime="2026-07-17 00:49:01.168942981 +0000 UTC m=+1935.745095552"
Jul 17 00:49:02.138381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:02.138342    2573 generic.go:356] "Generic (PLEG): container finished" podID="31583a0b-885d-4356-9bc1-c409819c7404" containerID="14f52cdf277c8efcb03d6d5f6687813fb0e2ee096e759f4de0581bba679829ed" exitCode=6
Jul 17 00:49:02.138869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:02.138428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" event={"ID":"31583a0b-885d-4356-9bc1-c409819c7404","Type":"ContainerDied","Data":"14f52cdf277c8efcb03d6d5f6687813fb0e2ee096e759f4de0581bba679829ed"}
Jul 17 00:49:02.138869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:02.138587    2573 scope.go:117] "RemoveContainer" containerID="14f52cdf277c8efcb03d6d5f6687813fb0e2ee096e759f4de0581bba679829ed"
Jul 17 00:49:03.145295 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:03.145261    2573 generic.go:356] "Generic (PLEG): container finished" podID="31583a0b-885d-4356-9bc1-c409819c7404" containerID="202c964e23782c001231d4d56781660a04436a899a35687d856df20f913d2422" exitCode=6
Jul 17 00:49:03.145701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:03.145360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" event={"ID":"31583a0b-885d-4356-9bc1-c409819c7404","Type":"ContainerDied","Data":"202c964e23782c001231d4d56781660a04436a899a35687d856df20f913d2422"}
Jul 17 00:49:03.145701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:03.145425    2573 scope.go:117] "RemoveContainer" containerID="14f52cdf277c8efcb03d6d5f6687813fb0e2ee096e759f4de0581bba679829ed"
Jul 17 00:49:03.145701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:03.145602    2573 scope.go:117] "RemoveContainer" containerID="202c964e23782c001231d4d56781660a04436a899a35687d856df20f913d2422"
Jul 17 00:49:03.145876 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:03.145844    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9_odh-ai-gateway-infra(31583a0b-885d-4356-9bc1-c409819c7404)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" podUID="31583a0b-885d-4356-9bc1-c409819c7404"
Jul 17 00:49:04.153021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:04.152989    2573 scope.go:117] "RemoveContainer" containerID="202c964e23782c001231d4d56781660a04436a899a35687d856df20f913d2422"
Jul 17 00:49:04.153515 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:04.153242    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9_odh-ai-gateway-infra(31583a0b-885d-4356-9bc1-c409819c7404)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" podUID="31583a0b-885d-4356-9bc1-c409819c7404"
Jul 17 00:49:12.641426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.641390    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"]
Jul 17 00:49:12.641869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.641751    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:12.641869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.641762    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:12.641869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.641842    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de60edd-1085-4ac5-a26f-5ab06e1bd586" containerName="revoke-keys"
Jul 17 00:49:12.645977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.645960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"
Jul 17 00:49:12.650287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.650251    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"]
Jul 17 00:49:12.707565 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.707536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780fd4bd-39f7-40d9-9969-2e444bf7d8d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l\" (UID: \"780fd4bd-39f7-40d9-9969-2e444bf7d8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"
Jul 17 00:49:12.808887 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.808854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780fd4bd-39f7-40d9-9969-2e444bf7d8d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l\" (UID: \"780fd4bd-39f7-40d9-9969-2e444bf7d8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"
Jul 17 00:49:12.809443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.809424    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780fd4bd-39f7-40d9-9969-2e444bf7d8d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l\" (UID: \"780fd4bd-39f7-40d9-9969-2e444bf7d8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"
Jul 17 00:49:12.958402 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.958332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"
Jul 17 00:49:12.962531 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:12.962505    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:49:12.962718 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:12.962696    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:49:13.090653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:13.090622    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"]
Jul 17 00:49:13.091978 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:49:13.091952    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780fd4bd_39f7_40d9_9969_2e444bf7d8d2.slice/crio-4373ac1c12a2917f03fc2d0471f1774e6aa0e1c89c576173ac9244cb915c4e19 WatchSource:0}: Error finding container 4373ac1c12a2917f03fc2d0471f1774e6aa0e1c89c576173ac9244cb915c4e19: Status 404 returned error can't find the container with id 4373ac1c12a2917f03fc2d0471f1774e6aa0e1c89c576173ac9244cb915c4e19
Jul 17 00:49:13.210713 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:13.210639    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" event={"ID":"780fd4bd-39f7-40d9-9969-2e444bf7d8d2","Type":"ContainerStarted","Data":"4373ac1c12a2917f03fc2d0471f1774e6aa0e1c89c576173ac9244cb915c4e19"}
Jul 17 00:49:14.217127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:14.217079    2573 generic.go:356] "Generic (PLEG): container finished" podID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerID="43d41c807552962c91f27e7ae26a8398e8260bb1534c42589c540762bccb11a3" exitCode=6
Jul 17 00:49:14.217127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:14.217130    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" event={"ID":"780fd4bd-39f7-40d9-9969-2e444bf7d8d2","Type":"ContainerDied","Data":"43d41c807552962c91f27e7ae26a8398e8260bb1534c42589c540762bccb11a3"}
Jul 17 00:49:14.217531 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:14.217329    2573 scope.go:117] "RemoveContainer" containerID="43d41c807552962c91f27e7ae26a8398e8260bb1534c42589c540762bccb11a3"
Jul 17 00:49:14.962722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:14.962697    2573 scope.go:117] "RemoveContainer" containerID="202c964e23782c001231d4d56781660a04436a899a35687d856df20f913d2422"
Jul 17 00:49:15.224649 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:15.224621    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" event={"ID":"31583a0b-885d-4356-9bc1-c409819c7404","Type":"ContainerStarted","Data":"8257bd2c68f523ecc9424c1fae4a5aebf64017cf7a8d0b039dc1dfd087f5b2a9"}
Jul 17 00:49:15.226487 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:15.226459    2573 generic.go:356] "Generic (PLEG): container finished" podID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerID="bd21a0d400a18f8358cb283d7d21dd57616d2e3b4cbda10b62cab00304d8a4c0" exitCode=6
Jul 17 00:49:15.226596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:15.226544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" event={"ID":"780fd4bd-39f7-40d9-9969-2e444bf7d8d2","Type":"ContainerDied","Data":"bd21a0d400a18f8358cb283d7d21dd57616d2e3b4cbda10b62cab00304d8a4c0"}
Jul 17 00:49:15.226650 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:15.226600    2573 scope.go:117] "RemoveContainer" containerID="43d41c807552962c91f27e7ae26a8398e8260bb1534c42589c540762bccb11a3"
Jul 17 00:49:15.226699 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:15.226686    2573 scope.go:117] "RemoveContainer" containerID="bd21a0d400a18f8358cb283d7d21dd57616d2e3b4cbda10b62cab00304d8a4c0"
Jul 17 00:49:15.226875 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:15.226855    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l_odh-ai-gateway-infra(780fd4bd-39f7-40d9-9969-2e444bf7d8d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2"
Jul 17 00:49:16.234093 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:16.234058    2573 generic.go:356] "Generic (PLEG): container finished" podID="31583a0b-885d-4356-9bc1-c409819c7404" containerID="8257bd2c68f523ecc9424c1fae4a5aebf64017cf7a8d0b039dc1dfd087f5b2a9" exitCode=6
Jul 17 00:49:16.234540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:16.234131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" event={"ID":"31583a0b-885d-4356-9bc1-c409819c7404","Type":"ContainerDied","Data":"8257bd2c68f523ecc9424c1fae4a5aebf64017cf7a8d0b039dc1dfd087f5b2a9"}
Jul 17 00:49:16.234540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:16.234207    2573 scope.go:117] "RemoveContainer" containerID="202c964e23782c001231d4d56781660a04436a899a35687d856df20f913d2422"
Jul 17 00:49:16.234540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:16.234511    2573 scope.go:117] "RemoveContainer" containerID="8257bd2c68f523ecc9424c1fae4a5aebf64017cf7a8d0b039dc1dfd087f5b2a9"
Jul 17 00:49:16.234773 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:16.234751    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9_odh-ai-gateway-infra(31583a0b-885d-4356-9bc1-c409819c7404)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" podUID="31583a0b-885d-4356-9bc1-c409819c7404"
Jul 17 00:49:16.235886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:16.235868    2573 scope.go:117] "RemoveContainer" containerID="bd21a0d400a18f8358cb283d7d21dd57616d2e3b4cbda10b62cab00304d8a4c0"
Jul 17 00:49:16.236067 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:16.236048    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l_odh-ai-gateway-infra(780fd4bd-39f7-40d9-9969-2e444bf7d8d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2"
Jul 17 00:49:16.247828 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:16.247805    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"]
Jul 17 00:49:17.378469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:17.378447    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"
Jul 17 00:49:17.440015 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:17.439988    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31583a0b-885d-4356-9bc1-c409819c7404-maas-api-service-ca\") pod \"31583a0b-885d-4356-9bc1-c409819c7404\" (UID: \"31583a0b-885d-4356-9bc1-c409819c7404\") "
Jul 17 00:49:17.440321 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:17.440299    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31583a0b-885d-4356-9bc1-c409819c7404-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31583a0b-885d-4356-9bc1-c409819c7404" (UID: "31583a0b-885d-4356-9bc1-c409819c7404"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:49:17.540522 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:17.540500    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31583a0b-885d-4356-9bc1-c409819c7404-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:49:18.249316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:18.249257    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"
Jul 17 00:49:18.249316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:18.249266    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9" event={"ID":"31583a0b-885d-4356-9bc1-c409819c7404","Type":"ContainerDied","Data":"ec62dc584fc9e6e81ef9360d56566ebcdd48da4cacc1b0167d61e5c1337b3064"}
Jul 17 00:49:18.249316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:18.249316    2573 scope.go:117] "RemoveContainer" containerID="8257bd2c68f523ecc9424c1fae4a5aebf64017cf7a8d0b039dc1dfd087f5b2a9"
Jul 17 00:49:18.268943 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:18.268919    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"]
Jul 17 00:49:18.271419 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:18.271395    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-zfgz9"]
Jul 17 00:49:19.968427 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:19.968392    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31583a0b-885d-4356-9bc1-c409819c7404" path="/var/lib/kubelet/pods/31583a0b-885d-4356-9bc1-c409819c7404/volumes"
Jul 17 00:49:23.962814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:23.962767    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:49:23.963350 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:23.963024    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:49:27.963257 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:27.963215    2573 scope.go:117] "RemoveContainer" containerID="bd21a0d400a18f8358cb283d7d21dd57616d2e3b4cbda10b62cab00304d8a4c0"
Jul 17 00:49:28.307019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:28.306983    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" event={"ID":"780fd4bd-39f7-40d9-9969-2e444bf7d8d2","Type":"ContainerStarted","Data":"45d78a08f94b70f7c8f9eaf55f567d3715b178660ebcba9de2afb647bc56a5ca"}
Jul 17 00:49:28.321744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:28.321687    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" podStartSLOduration=16.32167183 podStartE2EDuration="16.32167183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:49:28.319770906 +0000 UTC m=+1962.895923478" watchObservedRunningTime="2026-07-17 00:49:28.32167183 +0000 UTC m=+1962.897824403"
Jul 17 00:49:29.313881 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:29.313831    2573 generic.go:356] "Generic (PLEG): container finished" podID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerID="45d78a08f94b70f7c8f9eaf55f567d3715b178660ebcba9de2afb647bc56a5ca" exitCode=6
Jul 17 00:49:29.314390 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:29.313894    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" event={"ID":"780fd4bd-39f7-40d9-9969-2e444bf7d8d2","Type":"ContainerDied","Data":"45d78a08f94b70f7c8f9eaf55f567d3715b178660ebcba9de2afb647bc56a5ca"}
Jul 17 00:49:29.314390 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:29.313976    2573 scope.go:117] "RemoveContainer" containerID="bd21a0d400a18f8358cb283d7d21dd57616d2e3b4cbda10b62cab00304d8a4c0"
Jul 17 00:49:29.314390 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:29.314236    2573 scope.go:117] "RemoveContainer" containerID="45d78a08f94b70f7c8f9eaf55f567d3715b178660ebcba9de2afb647bc56a5ca"
Jul 17 00:49:29.314510 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:29.314448    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l_odh-ai-gateway-infra(780fd4bd-39f7-40d9-9969-2e444bf7d8d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2"
Jul 17 00:49:29.336344 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:29.336304    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"]
Jul 17 00:49:30.460562 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:30.460525    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"
Jul 17 00:49:30.556192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:30.556153    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780fd4bd-39f7-40d9-9969-2e444bf7d8d2-maas-api-service-ca\") pod \"780fd4bd-39f7-40d9-9969-2e444bf7d8d2\" (UID: \"780fd4bd-39f7-40d9-9969-2e444bf7d8d2\") "
Jul 17 00:49:30.556543 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:30.556517    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780fd4bd-39f7-40d9-9969-2e444bf7d8d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "780fd4bd-39f7-40d9-9969-2e444bf7d8d2" (UID: "780fd4bd-39f7-40d9-9969-2e444bf7d8d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:49:30.657747 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:30.657646    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780fd4bd-39f7-40d9-9969-2e444bf7d8d2-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:49:31.327636 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:31.327605    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"
Jul 17 00:49:31.327884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:31.327644    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l" event={"ID":"780fd4bd-39f7-40d9-9969-2e444bf7d8d2","Type":"ContainerDied","Data":"4373ac1c12a2917f03fc2d0471f1774e6aa0e1c89c576173ac9244cb915c4e19"}
Jul 17 00:49:31.327884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:31.327710    2573 scope.go:117] "RemoveContainer" containerID="45d78a08f94b70f7c8f9eaf55f567d3715b178660ebcba9de2afb647bc56a5ca"
Jul 17 00:49:31.349876 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:31.349839    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"]
Jul 17 00:49:31.355216 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:31.355188    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-jdm8l"]
Jul 17 00:49:31.968737 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:31.968699    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" path="/var/lib/kubelet/pods/780fd4bd-39f7-40d9-9969-2e444bf7d8d2/volumes"
Jul 17 00:49:37.963200 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:37.963102    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:49:37.963603 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:37.963374    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:49:38.485311 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485266    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"]
Jul 17 00:49:38.485720 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485701    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:38.485844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485722    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:38.485844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485737    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:38.485844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485747    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:38.485844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485802    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:38.485844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485812    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:38.485844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485827    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:38.485844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485836    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:38.486212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485936    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:38.486212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485951    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:38.486212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485960    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:38.486212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485969    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:38.486212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.485981    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:38.490562 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.490529    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"
Jul 17 00:49:38.492537 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.492503    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:49:38.494612 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.494577    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"]
Jul 17 00:49:38.533335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.533291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff526d7-3946-4b04-8927-12d75d8667cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq\" (UID: \"cff526d7-3946-4b04-8927-12d75d8667cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"
Jul 17 00:49:38.634636 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.634590    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff526d7-3946-4b04-8927-12d75d8667cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq\" (UID: \"cff526d7-3946-4b04-8927-12d75d8667cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"
Jul 17 00:49:38.635323 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.635302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff526d7-3946-4b04-8927-12d75d8667cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq\" (UID: \"cff526d7-3946-4b04-8927-12d75d8667cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"
Jul 17 00:49:38.804250 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.804214    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"
Jul 17 00:49:38.950706 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:38.950677    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"]
Jul 17 00:49:38.952763 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:49:38.952733    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff526d7_3946_4b04_8927_12d75d8667cf.slice/crio-d7299270ea5e5d312684a591cb38a98420db104d126b26adb0a9917d00a97444 WatchSource:0}: Error finding container d7299270ea5e5d312684a591cb38a98420db104d126b26adb0a9917d00a97444: Status 404 returned error can't find the container with id d7299270ea5e5d312684a591cb38a98420db104d126b26adb0a9917d00a97444
Jul 17 00:49:39.373676 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:39.373635    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" event={"ID":"cff526d7-3946-4b04-8927-12d75d8667cf","Type":"ContainerStarted","Data":"100cbd1c978f1efbed40a1a838a31e15475f45e3935c9517d46ba0a046a50830"}
Jul 17 00:49:39.374146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:39.373702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" event={"ID":"cff526d7-3946-4b04-8927-12d75d8667cf","Type":"ContainerStarted","Data":"d7299270ea5e5d312684a591cb38a98420db104d126b26adb0a9917d00a97444"}
Jul 17 00:49:39.388993 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:39.388939    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" podStartSLOduration=1.388919957 podStartE2EDuration="1.388919957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:49:39.387171534 +0000 UTC m=+1973.963324110" watchObservedRunningTime="2026-07-17 00:49:39.388919957 +0000 UTC m=+1973.965072572"
Jul 17 00:49:40.382333 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:40.382294    2573 generic.go:356] "Generic (PLEG): container finished" podID="cff526d7-3946-4b04-8927-12d75d8667cf" containerID="100cbd1c978f1efbed40a1a838a31e15475f45e3935c9517d46ba0a046a50830" exitCode=7
Jul 17 00:49:40.382748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:40.382339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" event={"ID":"cff526d7-3946-4b04-8927-12d75d8667cf","Type":"ContainerDied","Data":"100cbd1c978f1efbed40a1a838a31e15475f45e3935c9517d46ba0a046a50830"}
Jul 17 00:49:40.382748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:40.382550    2573 scope.go:117] "RemoveContainer" containerID="100cbd1c978f1efbed40a1a838a31e15475f45e3935c9517d46ba0a046a50830"
Jul 17 00:49:41.389233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:41.389198    2573 generic.go:356] "Generic (PLEG): container finished" podID="cff526d7-3946-4b04-8927-12d75d8667cf" containerID="5e8723795f1c171c777e922323211b638b86b355b4d955b824aaa603711f5acf" exitCode=7
Jul 17 00:49:41.389690 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:41.389281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" event={"ID":"cff526d7-3946-4b04-8927-12d75d8667cf","Type":"ContainerDied","Data":"5e8723795f1c171c777e922323211b638b86b355b4d955b824aaa603711f5acf"}
Jul 17 00:49:41.389690 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:41.389334    2573 scope.go:117] "RemoveContainer" containerID="100cbd1c978f1efbed40a1a838a31e15475f45e3935c9517d46ba0a046a50830"
Jul 17 00:49:41.389690 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:41.389523    2573 scope.go:117] "RemoveContainer" containerID="5e8723795f1c171c777e922323211b638b86b355b4d955b824aaa603711f5acf"
Jul 17 00:49:41.389837 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:41.389753    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq_odh-ai-gateway-infra(cff526d7-3946-4b04-8927-12d75d8667cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" podUID="cff526d7-3946-4b04-8927-12d75d8667cf"
Jul 17 00:49:42.396737 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:42.396703    2573 scope.go:117] "RemoveContainer" containerID="5e8723795f1c171c777e922323211b638b86b355b4d955b824aaa603711f5acf"
Jul 17 00:49:42.397265 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:42.396989    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq_odh-ai-gateway-infra(cff526d7-3946-4b04-8927-12d75d8667cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" podUID="cff526d7-3946-4b04-8927-12d75d8667cf"
Jul 17 00:49:50.237604 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.237564    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"]
Jul 17 00:49:50.238034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.237965    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:50.238034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.237978    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="31583a0b-885d-4356-9bc1-c409819c7404" containerName="revoke-keys"
Jul 17 00:49:50.238034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.237994    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:50.238034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.238000    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:50.238172 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.238070    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="780fd4bd-39f7-40d9-9969-2e444bf7d8d2" containerName="revoke-keys"
Jul 17 00:49:50.241001 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.240982    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"
Jul 17 00:49:50.247138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.247113    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"]
Jul 17 00:49:50.337825 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.337759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1800c06b-ffc4-41fb-8886-dd84951745df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn\" (UID: \"1800c06b-ffc4-41fb-8886-dd84951745df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"
Jul 17 00:49:50.438543 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.438503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1800c06b-ffc4-41fb-8886-dd84951745df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn\" (UID: \"1800c06b-ffc4-41fb-8886-dd84951745df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"
Jul 17 00:49:50.439199 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.439177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1800c06b-ffc4-41fb-8886-dd84951745df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn\" (UID: \"1800c06b-ffc4-41fb-8886-dd84951745df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"
Jul 17 00:49:50.555265 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.555216    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"
Jul 17 00:49:50.896187 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:50.896146    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"]
Jul 17 00:49:50.898016 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:49:50.897975    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1800c06b_ffc4_41fb_8886_dd84951745df.slice/crio-07021c195128fb09cd7d9fec90c4354452fdbbce3ab0c3bfbcda3aeee9657b89 WatchSource:0}: Error finding container 07021c195128fb09cd7d9fec90c4354452fdbbce3ab0c3bfbcda3aeee9657b89: Status 404 returned error can't find the container with id 07021c195128fb09cd7d9fec90c4354452fdbbce3ab0c3bfbcda3aeee9657b89
Jul 17 00:49:51.446054 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:51.446017    2573 generic.go:356] "Generic (PLEG): container finished" podID="1800c06b-ffc4-41fb-8886-dd84951745df" containerID="a95f0f26f148203ec19a911a71a58a4dbac33a21562a2dedb238fe182c9bb355" exitCode=6
Jul 17 00:49:51.446470 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:51.446111    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" event={"ID":"1800c06b-ffc4-41fb-8886-dd84951745df","Type":"ContainerDied","Data":"a95f0f26f148203ec19a911a71a58a4dbac33a21562a2dedb238fe182c9bb355"}
Jul 17 00:49:51.446470 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:51.446183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" event={"ID":"1800c06b-ffc4-41fb-8886-dd84951745df","Type":"ContainerStarted","Data":"07021c195128fb09cd7d9fec90c4354452fdbbce3ab0c3bfbcda3aeee9657b89"}
Jul 17 00:49:51.446470 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:51.446258    2573 scope.go:117] "RemoveContainer" containerID="a95f0f26f148203ec19a911a71a58a4dbac33a21562a2dedb238fe182c9bb355"
Jul 17 00:49:52.453028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:52.452940    2573 generic.go:356] "Generic (PLEG): container finished" podID="1800c06b-ffc4-41fb-8886-dd84951745df" containerID="add7d722919592cd62e2871120e4c2ab03e2c9afbbeccf59747cd4c6e06fd913" exitCode=6
Jul 17 00:49:52.453028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:52.453013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" event={"ID":"1800c06b-ffc4-41fb-8886-dd84951745df","Type":"ContainerDied","Data":"add7d722919592cd62e2871120e4c2ab03e2c9afbbeccf59747cd4c6e06fd913"}
Jul 17 00:49:52.453469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:52.453098    2573 scope.go:117] "RemoveContainer" containerID="a95f0f26f148203ec19a911a71a58a4dbac33a21562a2dedb238fe182c9bb355"
Jul 17 00:49:52.453469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:52.453280    2573 scope.go:117] "RemoveContainer" containerID="add7d722919592cd62e2871120e4c2ab03e2c9afbbeccf59747cd4c6e06fd913"
Jul 17 00:49:52.453566 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:52.453543    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn_odh-ai-gateway-infra(1800c06b-ffc4-41fb-8886-dd84951745df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" podUID="1800c06b-ffc4-41fb-8886-dd84951745df"
Jul 17 00:49:52.962697 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:52.962660    2573 scope.go:117] "RemoveContainer" containerID="5e8723795f1c171c777e922323211b638b86b355b4d955b824aaa603711f5acf"
Jul 17 00:49:52.962929 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:52.962773    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:49:53.459862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:53.459825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" event={"ID":"cff526d7-3946-4b04-8927-12d75d8667cf","Type":"ContainerStarted","Data":"26e6bc55f48f7904b1f91b3d6c1522448a27cd1de1755403adf8cd5d905b9edd"}
Jul 17 00:49:53.461664 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:53.461633    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerStarted","Data":"406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"}
Jul 17 00:49:53.461879 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:53.461853    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:49:53.463335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:53.463319    2573 scope.go:117] "RemoveContainer" containerID="add7d722919592cd62e2871120e4c2ab03e2c9afbbeccf59747cd4c6e06fd913"
Jul 17 00:49:53.463481 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:53.463463    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn_odh-ai-gateway-infra(1800c06b-ffc4-41fb-8886-dd84951745df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" podUID="1800c06b-ffc4-41fb-8886-dd84951745df"
Jul 17 00:49:53.490264 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:53.490210    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podStartSLOduration=187.565851608 podStartE2EDuration="3m10.490194024s" totalImagesPullingTime="2.924342416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:46:43 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:49:53.487762023 +0000 UTC m=+1988.063914612" watchObservedRunningTime="2026-07-17 00:49:53.490194024 +0000 UTC m=+1988.066346597"
Jul 17 00:49:54.470745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.470713    2573 generic.go:356] "Generic (PLEG): container finished" podID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51" exitCode=1
Jul 17 00:49:54.471247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.470765    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"}
Jul 17 00:49:54.471247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.470855    2573 scope.go:117] "RemoveContainer" containerID="3c56d5c0d36a2b3dab8b4159153c60b5bdcd2ee27078e8952fc4cb728f08273a"
Jul 17 00:49:54.471247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.471098    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:49:54.471412 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:54.471330    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:49:54.472916 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.472891    2573 generic.go:356] "Generic (PLEG): container finished" podID="cff526d7-3946-4b04-8927-12d75d8667cf" containerID="26e6bc55f48f7904b1f91b3d6c1522448a27cd1de1755403adf8cd5d905b9edd" exitCode=7
Jul 17 00:49:54.473021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.472940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" event={"ID":"cff526d7-3946-4b04-8927-12d75d8667cf","Type":"ContainerDied","Data":"26e6bc55f48f7904b1f91b3d6c1522448a27cd1de1755403adf8cd5d905b9edd"}
Jul 17 00:49:54.473163 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.473151    2573 scope.go:117] "RemoveContainer" containerID="26e6bc55f48f7904b1f91b3d6c1522448a27cd1de1755403adf8cd5d905b9edd"
Jul 17 00:49:54.473322 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:54.473307    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq_odh-ai-gateway-infra(cff526d7-3946-4b04-8927-12d75d8667cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" podUID="cff526d7-3946-4b04-8927-12d75d8667cf"
Jul 17 00:49:54.482866 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.482845    2573 scope.go:117] "RemoveContainer" containerID="5e8723795f1c171c777e922323211b638b86b355b4d955b824aaa603711f5acf"
Jul 17 00:49:54.484128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:54.484107    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"]
Jul 17 00:49:55.482994 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:55.482966    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:49:55.483305 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:49:55.483202    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:49:55.622648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:55.622623    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"
Jul 17 00:49:55.784345 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:55.784308    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff526d7-3946-4b04-8927-12d75d8667cf-maas-api-service-ca\") pod \"cff526d7-3946-4b04-8927-12d75d8667cf\" (UID: \"cff526d7-3946-4b04-8927-12d75d8667cf\") "
Jul 17 00:49:55.784697 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:55.784678    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff526d7-3946-4b04-8927-12d75d8667cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cff526d7-3946-4b04-8927-12d75d8667cf" (UID: "cff526d7-3946-4b04-8927-12d75d8667cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:49:55.885349 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:55.885309    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff526d7-3946-4b04-8927-12d75d8667cf-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:49:56.490499 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:56.490412    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"
Jul 17 00:49:56.490937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:56.490525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq" event={"ID":"cff526d7-3946-4b04-8927-12d75d8667cf","Type":"ContainerDied","Data":"d7299270ea5e5d312684a591cb38a98420db104d126b26adb0a9917d00a97444"}
Jul 17 00:49:56.490937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:56.490610    2573 scope.go:117] "RemoveContainer" containerID="26e6bc55f48f7904b1f91b3d6c1522448a27cd1de1755403adf8cd5d905b9edd"
Jul 17 00:49:56.507008 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:56.506984    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"]
Jul 17 00:49:56.510653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:56.510631    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vxrrq"]
Jul 17 00:49:57.968571 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:49:57.968538    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" path="/var/lib/kubelet/pods/cff526d7-3946-4b04-8927-12d75d8667cf/volumes"
Jul 17 00:50:02.686769 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.686734    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"]
Jul 17 00:50:02.687366 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687317    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.687366 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687338    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.687366 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687362    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.687539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687371    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.687539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687380    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.687539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687388    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.687539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687526    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.687539 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.687539    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:02.692183 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.692164    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"
Jul 17 00:50:02.697223 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.697194    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"]
Jul 17 00:50:02.847034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.847002    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cc72a6-3e78-44d2-a7d6-46c978c376ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp\" (UID: \"99cc72a6-3e78-44d2-a7d6-46c978c376ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"
Jul 17 00:50:02.948292 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.948209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cc72a6-3e78-44d2-a7d6-46c978c376ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp\" (UID: \"99cc72a6-3e78-44d2-a7d6-46c978c376ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"
Jul 17 00:50:02.948863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:02.948842    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cc72a6-3e78-44d2-a7d6-46c978c376ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp\" (UID: \"99cc72a6-3e78-44d2-a7d6-46c978c376ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"
Jul 17 00:50:03.004304 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:03.004276    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"
Jul 17 00:50:03.362412 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:03.362383    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"]
Jul 17 00:50:03.363967 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:50:03.363943    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cc72a6_3e78_44d2_a7d6_46c978c376ff.slice/crio-ffe5ed3a4f0dc8fdc862055c52a5d587274f3145a3d4db839f7fd802d2c05207 WatchSource:0}: Error finding container ffe5ed3a4f0dc8fdc862055c52a5d587274f3145a3d4db839f7fd802d2c05207: Status 404 returned error can't find the container with id ffe5ed3a4f0dc8fdc862055c52a5d587274f3145a3d4db839f7fd802d2c05207
Jul 17 00:50:03.528388 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:03.528348    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" event={"ID":"99cc72a6-3e78-44d2-a7d6-46c978c376ff","Type":"ContainerStarted","Data":"ffe5ed3a4f0dc8fdc862055c52a5d587274f3145a3d4db839f7fd802d2c05207"}
Jul 17 00:50:03.963008 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:03.962983    2573 scope.go:117] "RemoveContainer" containerID="add7d722919592cd62e2871120e4c2ab03e2c9afbbeccf59747cd4c6e06fd913"
Jul 17 00:50:04.056619 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:04.056594    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:50:04.057067 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:04.057052    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:50:04.057285 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:04.057256    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:50:04.535999 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:04.535961    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" event={"ID":"1800c06b-ffc4-41fb-8886-dd84951745df","Type":"ContainerStarted","Data":"db850dfc1f33175e2c15bd23a4d021baf2bcfbefa87762eda0e75f58b1025147"}
Jul 17 00:50:04.537410 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:04.537381    2573 generic.go:356] "Generic (PLEG): container finished" podID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerID="6b7035770c995a6684a2d7de2a1b30a8a63a0fcf00174e3ba152b70728d9dd22" exitCode=6
Jul 17 00:50:04.537548 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:04.537471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" event={"ID":"99cc72a6-3e78-44d2-a7d6-46c978c376ff","Type":"ContainerDied","Data":"6b7035770c995a6684a2d7de2a1b30a8a63a0fcf00174e3ba152b70728d9dd22"}
Jul 17 00:50:04.537613 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:04.537599    2573 scope.go:117] "RemoveContainer" containerID="6b7035770c995a6684a2d7de2a1b30a8a63a0fcf00174e3ba152b70728d9dd22"
Jul 17 00:50:04.549856 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:04.549808    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" podStartSLOduration=14.549770018 podStartE2EDuration="14.549770018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:50:04.549317465 +0000 UTC m=+1999.125470043" watchObservedRunningTime="2026-07-17 00:50:04.549770018 +0000 UTC m=+1999.125922593"
Jul 17 00:50:05.544460 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.544425    2573 generic.go:356] "Generic (PLEG): container finished" podID="1800c06b-ffc4-41fb-8886-dd84951745df" containerID="db850dfc1f33175e2c15bd23a4d021baf2bcfbefa87762eda0e75f58b1025147" exitCode=6
Jul 17 00:50:05.544967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.544502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" event={"ID":"1800c06b-ffc4-41fb-8886-dd84951745df","Type":"ContainerDied","Data":"db850dfc1f33175e2c15bd23a4d021baf2bcfbefa87762eda0e75f58b1025147"}
Jul 17 00:50:05.544967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.544564    2573 scope.go:117] "RemoveContainer" containerID="add7d722919592cd62e2871120e4c2ab03e2c9afbbeccf59747cd4c6e06fd913"
Jul 17 00:50:05.544967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.544862    2573 scope.go:117] "RemoveContainer" containerID="db850dfc1f33175e2c15bd23a4d021baf2bcfbefa87762eda0e75f58b1025147"
Jul 17 00:50:05.545117 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:05.545097    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn_odh-ai-gateway-infra(1800c06b-ffc4-41fb-8886-dd84951745df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" podUID="1800c06b-ffc4-41fb-8886-dd84951745df"
Jul 17 00:50:05.546501 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.546478    2573 generic.go:356] "Generic (PLEG): container finished" podID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerID="4c70d689cf9624b96a27723e9fc03ccd4711cb9ea554bc3215e26e2fd206b5eb" exitCode=6
Jul 17 00:50:05.546609 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.546555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" event={"ID":"99cc72a6-3e78-44d2-a7d6-46c978c376ff","Type":"ContainerDied","Data":"4c70d689cf9624b96a27723e9fc03ccd4711cb9ea554bc3215e26e2fd206b5eb"}
Jul 17 00:50:05.546758 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.546745    2573 scope.go:117] "RemoveContainer" containerID="4c70d689cf9624b96a27723e9fc03ccd4711cb9ea554bc3215e26e2fd206b5eb"
Jul 17 00:50:05.546974 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:05.546959    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp_odh-ai-gateway-infra(99cc72a6-3e78-44d2-a7d6-46c978c376ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff"
Jul 17 00:50:05.555801 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.555774    2573 scope.go:117] "RemoveContainer" containerID="6b7035770c995a6684a2d7de2a1b30a8a63a0fcf00174e3ba152b70728d9dd22"
Jul 17 00:50:05.561343 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:05.561323    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"]
Jul 17 00:50:06.554854 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:06.554832    2573 scope.go:117] "RemoveContainer" containerID="4c70d689cf9624b96a27723e9fc03ccd4711cb9ea554bc3215e26e2fd206b5eb"
Jul 17 00:50:06.555175 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:06.555021    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp_odh-ai-gateway-infra(99cc72a6-3e78-44d2-a7d6-46c978c376ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff"
Jul 17 00:50:06.687385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:06.687360    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"
Jul 17 00:50:06.778958 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:06.778924    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1800c06b-ffc4-41fb-8886-dd84951745df-maas-api-service-ca\") pod \"1800c06b-ffc4-41fb-8886-dd84951745df\" (UID: \"1800c06b-ffc4-41fb-8886-dd84951745df\") "
Jul 17 00:50:06.779291 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:06.779271    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1800c06b-ffc4-41fb-8886-dd84951745df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1800c06b-ffc4-41fb-8886-dd84951745df" (UID: "1800c06b-ffc4-41fb-8886-dd84951745df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:50:06.880187 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:06.880107    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1800c06b-ffc4-41fb-8886-dd84951745df-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:50:07.561039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:07.560994    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn" event={"ID":"1800c06b-ffc4-41fb-8886-dd84951745df","Type":"ContainerDied","Data":"07021c195128fb09cd7d9fec90c4354452fdbbce3ab0c3bfbcda3aeee9657b89"}
Jul 17 00:50:07.561039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:07.561015    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"
Jul 17 00:50:07.561620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:07.561060    2573 scope.go:117] "RemoveContainer" containerID="db850dfc1f33175e2c15bd23a4d021baf2bcfbefa87762eda0e75f58b1025147"
Jul 17 00:50:07.583939 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:07.583910    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"]
Jul 17 00:50:07.589426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:07.589406    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rkzwn"]
Jul 17 00:50:07.968338 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:07.968260    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" path="/var/lib/kubelet/pods/1800c06b-ffc4-41fb-8886-dd84951745df/volumes"
Jul 17 00:50:14.962534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:14.962505    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:50:14.962932 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:14.962660    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:50:16.962290 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:16.962261    2573 scope.go:117] "RemoveContainer" containerID="4c70d689cf9624b96a27723e9fc03ccd4711cb9ea554bc3215e26e2fd206b5eb"
Jul 17 00:50:17.620198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:17.620169    2573 generic.go:356] "Generic (PLEG): container finished" podID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerID="c4d9639f78fee73e055a21d92d7cd14fa46064f02334ef89be79ee60be5707c3" exitCode=6
Jul 17 00:50:17.620299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:17.620232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" event={"ID":"99cc72a6-3e78-44d2-a7d6-46c978c376ff","Type":"ContainerDied","Data":"c4d9639f78fee73e055a21d92d7cd14fa46064f02334ef89be79ee60be5707c3"}
Jul 17 00:50:17.620299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:17.620284    2573 scope.go:117] "RemoveContainer" containerID="4c70d689cf9624b96a27723e9fc03ccd4711cb9ea554bc3215e26e2fd206b5eb"
Jul 17 00:50:17.620582 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:17.620564    2573 scope.go:117] "RemoveContainer" containerID="c4d9639f78fee73e055a21d92d7cd14fa46064f02334ef89be79ee60be5707c3"
Jul 17 00:50:17.620785 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:17.620764    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp_odh-ai-gateway-infra(99cc72a6-3e78-44d2-a7d6-46c978c376ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff"
Jul 17 00:50:18.646106 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:18.646077    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"]
Jul 17 00:50:18.772369 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:18.772344    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"
Jul 17 00:50:18.869521 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:18.869493    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cc72a6-3e78-44d2-a7d6-46c978c376ff-maas-api-service-ca\") pod \"99cc72a6-3e78-44d2-a7d6-46c978c376ff\" (UID: \"99cc72a6-3e78-44d2-a7d6-46c978c376ff\") "
Jul 17 00:50:18.869855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:18.869835    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cc72a6-3e78-44d2-a7d6-46c978c376ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99cc72a6-3e78-44d2-a7d6-46c978c376ff" (UID: "99cc72a6-3e78-44d2-a7d6-46c978c376ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:50:18.970691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:18.970633    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cc72a6-3e78-44d2-a7d6-46c978c376ff-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:50:19.632985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:19.632955    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"
Jul 17 00:50:19.633196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:19.632999    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp" event={"ID":"99cc72a6-3e78-44d2-a7d6-46c978c376ff","Type":"ContainerDied","Data":"ffe5ed3a4f0dc8fdc862055c52a5d587274f3145a3d4db839f7fd802d2c05207"}
Jul 17 00:50:19.633196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:19.633087    2573 scope.go:117] "RemoveContainer" containerID="c4d9639f78fee73e055a21d92d7cd14fa46064f02334ef89be79ee60be5707c3"
Jul 17 00:50:19.653959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:19.653934    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"]
Jul 17 00:50:19.657563 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:19.657540    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nfxpp"]
Jul 17 00:50:19.967663 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:19.967585    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" path="/var/lib/kubelet/pods/99cc72a6-3e78-44d2-a7d6-46c978c376ff/volumes"
Jul 17 00:50:26.962410 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:26.962376    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:50:26.962905 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:26.962539    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:50:28.441825 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.441775    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"]
Jul 17 00:50:28.442315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442297    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.442398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442319    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.442398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442335    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.442398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442343    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.442398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442363    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.442398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442372    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.442398 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442395    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442405    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442418    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442427    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442438    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442446    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442527    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff526d7-3946-4b04-8927-12d75d8667cf" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442538    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442546    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442556    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442566    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="99cc72a6-3e78-44d2-a7d6-46c978c376ff" containerName="revoke-keys"
Jul 17 00:50:28.442687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.442577    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:28.446982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.446959    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"
Jul 17 00:50:28.449163 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.449142    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:50:28.451493 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.451466    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"]
Jul 17 00:50:28.542553 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.542522    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7c4dc-745d-4475-ba7d-2f2ef3f875a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-8mz27\" (UID: \"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"
Jul 17 00:50:28.643201 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.643169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7c4dc-745d-4475-ba7d-2f2ef3f875a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-8mz27\" (UID: \"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"
Jul 17 00:50:28.643744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.643726    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7c4dc-745d-4475-ba7d-2f2ef3f875a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-8mz27\" (UID: \"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"
Jul 17 00:50:28.757827 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.757784    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"
Jul 17 00:50:28.900156 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:28.900123    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"]
Jul 17 00:50:28.901383 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:50:28.901360    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f7c4dc_745d_4475_ba7d_2f2ef3f875a4.slice/crio-ba5495634e001a0f8bc5047106eecf4c0416d34d17df6fef3a040716840a3472 WatchSource:0}: Error finding container ba5495634e001a0f8bc5047106eecf4c0416d34d17df6fef3a040716840a3472: Status 404 returned error can't find the container with id ba5495634e001a0f8bc5047106eecf4c0416d34d17df6fef3a040716840a3472
Jul 17 00:50:29.688643 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:29.688609    2573 generic.go:356] "Generic (PLEG): container finished" podID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerID="5f294c695b45d213c54eefe3ff02aa5c578a44c12e98608fcbbcf4fcba24df54" exitCode=7
Jul 17 00:50:29.689065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:29.688651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" event={"ID":"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4","Type":"ContainerDied","Data":"5f294c695b45d213c54eefe3ff02aa5c578a44c12e98608fcbbcf4fcba24df54"}
Jul 17 00:50:29.689065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:29.688694    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" event={"ID":"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4","Type":"ContainerStarted","Data":"ba5495634e001a0f8bc5047106eecf4c0416d34d17df6fef3a040716840a3472"}
Jul 17 00:50:29.689065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:29.688874    2573 scope.go:117] "RemoveContainer" containerID="5f294c695b45d213c54eefe3ff02aa5c578a44c12e98608fcbbcf4fcba24df54"
Jul 17 00:50:30.702883 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:30.702846    2573 generic.go:356] "Generic (PLEG): container finished" podID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerID="1d49a7751c5417e0e89845f59466e6675f6e118e83495db31927938a16899e4b" exitCode=7
Jul 17 00:50:30.703385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:30.702931    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" event={"ID":"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4","Type":"ContainerDied","Data":"1d49a7751c5417e0e89845f59466e6675f6e118e83495db31927938a16899e4b"}
Jul 17 00:50:30.703385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:30.703051    2573 scope.go:117] "RemoveContainer" containerID="5f294c695b45d213c54eefe3ff02aa5c578a44c12e98608fcbbcf4fcba24df54"
Jul 17 00:50:30.703385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:30.703311    2573 scope.go:117] "RemoveContainer" containerID="1d49a7751c5417e0e89845f59466e6675f6e118e83495db31927938a16899e4b"
Jul 17 00:50:30.703578 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:30.703559    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-8mz27_odh-ai-gateway-infra(37f7c4dc-745d-4475-ba7d-2f2ef3f875a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4"
Jul 17 00:50:31.710613 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:31.710583    2573 scope.go:117] "RemoveContainer" containerID="1d49a7751c5417e0e89845f59466e6675f6e118e83495db31927938a16899e4b"
Jul 17 00:50:31.711036 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:31.710857    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-8mz27_odh-ai-gateway-infra(37f7c4dc-745d-4475-ba7d-2f2ef3f875a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4"
Jul 17 00:50:37.963153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:37.963111    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:50:37.963639 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:37.963279    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:50:40.298863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.298821    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"]
Jul 17 00:50:40.299276 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.299262    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1800c06b-ffc4-41fb-8886-dd84951745df" containerName="revoke-keys"
Jul 17 00:50:40.303814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.303774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"
Jul 17 00:50:40.314870 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.314837    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"]
Jul 17 00:50:40.347518 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.347471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8444deb2-dfb7-4c93-92f6-aa55a4ff54cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6987b\" (UID: \"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"
Jul 17 00:50:40.448159 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.448112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8444deb2-dfb7-4c93-92f6-aa55a4ff54cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6987b\" (UID: \"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"
Jul 17 00:50:40.448765 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.448742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8444deb2-dfb7-4c93-92f6-aa55a4ff54cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6987b\" (UID: \"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"
Jul 17 00:50:40.615476 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.615377    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"
Jul 17 00:50:40.962488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:40.962456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"]
Jul 17 00:50:40.964341 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:50:40.964316    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8444deb2_dfb7_4c93_92f6_aa55a4ff54cb.slice/crio-8ff55a91ba3ad7ea2315ef2086db106fbe2267d819c5cfcdca327239508e6183 WatchSource:0}: Error finding container 8ff55a91ba3ad7ea2315ef2086db106fbe2267d819c5cfcdca327239508e6183: Status 404 returned error can't find the container with id 8ff55a91ba3ad7ea2315ef2086db106fbe2267d819c5cfcdca327239508e6183
Jul 17 00:50:41.766745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:41.766708    2573 generic.go:356] "Generic (PLEG): container finished" podID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerID="9fdc87176945da8d6d1d75cec4591020384138fe802cfe7cc130d998e36d9be3" exitCode=6
Jul 17 00:50:41.767165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:41.766751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" event={"ID":"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb","Type":"ContainerDied","Data":"9fdc87176945da8d6d1d75cec4591020384138fe802cfe7cc130d998e36d9be3"}
Jul 17 00:50:41.767165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:41.766844    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" event={"ID":"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb","Type":"ContainerStarted","Data":"8ff55a91ba3ad7ea2315ef2086db106fbe2267d819c5cfcdca327239508e6183"}
Jul 17 00:50:41.767165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:41.767030    2573 scope.go:117] "RemoveContainer" containerID="9fdc87176945da8d6d1d75cec4591020384138fe802cfe7cc130d998e36d9be3"
Jul 17 00:50:42.773682 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:42.773647    2573 generic.go:356] "Generic (PLEG): container finished" podID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerID="ea79f4ae31b40a385646388f86b30fbd66b39d06f521aa7a0f52a18bd476d2ad" exitCode=6
Jul 17 00:50:42.774168 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:42.773726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" event={"ID":"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb","Type":"ContainerDied","Data":"ea79f4ae31b40a385646388f86b30fbd66b39d06f521aa7a0f52a18bd476d2ad"}
Jul 17 00:50:42.774168 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:42.773809    2573 scope.go:117] "RemoveContainer" containerID="9fdc87176945da8d6d1d75cec4591020384138fe802cfe7cc130d998e36d9be3"
Jul 17 00:50:42.774168 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:42.774029    2573 scope.go:117] "RemoveContainer" containerID="ea79f4ae31b40a385646388f86b30fbd66b39d06f521aa7a0f52a18bd476d2ad"
Jul 17 00:50:42.774351 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:42.774241    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6987b_odh-ai-gateway-infra(8444deb2-dfb7-4c93-92f6-aa55a4ff54cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb"
Jul 17 00:50:43.781130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:43.781096    2573 scope.go:117] "RemoveContainer" containerID="ea79f4ae31b40a385646388f86b30fbd66b39d06f521aa7a0f52a18bd476d2ad"
Jul 17 00:50:43.781593 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:43.781371    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6987b_odh-ai-gateway-infra(8444deb2-dfb7-4c93-92f6-aa55a4ff54cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb"
Jul 17 00:50:44.962725 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:44.962689    2573 scope.go:117] "RemoveContainer" containerID="1d49a7751c5417e0e89845f59466e6675f6e118e83495db31927938a16899e4b"
Jul 17 00:50:45.795277 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:45.795185    2573 generic.go:356] "Generic (PLEG): container finished" podID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerID="3d7b83a25e512a8bab1fc698fa7293a7aec0214a22340d2d54487f353c8c0259" exitCode=7
Jul 17 00:50:45.795277 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:45.795265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" event={"ID":"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4","Type":"ContainerDied","Data":"3d7b83a25e512a8bab1fc698fa7293a7aec0214a22340d2d54487f353c8c0259"}
Jul 17 00:50:45.795493 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:45.795336    2573 scope.go:117] "RemoveContainer" containerID="1d49a7751c5417e0e89845f59466e6675f6e118e83495db31927938a16899e4b"
Jul 17 00:50:45.795675 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:45.795645    2573 scope.go:117] "RemoveContainer" containerID="3d7b83a25e512a8bab1fc698fa7293a7aec0214a22340d2d54487f353c8c0259"
Jul 17 00:50:45.795977 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:45.795944    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-8mz27_odh-ai-gateway-infra(37f7c4dc-745d-4475-ba7d-2f2ef3f875a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4"
Jul 17 00:50:46.817740 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:46.817705    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"]
Jul 17 00:50:46.955026 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:46.955001    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"
Jul 17 00:50:47.005694 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.005662    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7c4dc-745d-4475-ba7d-2f2ef3f875a4-maas-api-service-ca\") pod \"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4\" (UID: \"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4\") "
Jul 17 00:50:47.006073 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.006049    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f7c4dc-745d-4475-ba7d-2f2ef3f875a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" (UID: "37f7c4dc-745d-4475-ba7d-2f2ef3f875a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:50:47.107154 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.107049    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7c4dc-745d-4475-ba7d-2f2ef3f875a4-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:50:47.809537 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.809504    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"
Jul 17 00:50:47.809537 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.809518    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27" event={"ID":"37f7c4dc-745d-4475-ba7d-2f2ef3f875a4","Type":"ContainerDied","Data":"ba5495634e001a0f8bc5047106eecf4c0416d34d17df6fef3a040716840a3472"}
Jul 17 00:50:47.809783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.809583    2573 scope.go:117] "RemoveContainer" containerID="3d7b83a25e512a8bab1fc698fa7293a7aec0214a22340d2d54487f353c8c0259"
Jul 17 00:50:47.835445 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.835413    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"]
Jul 17 00:50:47.837783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.837755    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-8mz27"]
Jul 17 00:50:47.968655 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:47.968612    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" path="/var/lib/kubelet/pods/37f7c4dc-745d-4475-ba7d-2f2ef3f875a4/volumes"
Jul 17 00:50:50.962157 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:50.962125    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:50:50.962604 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:50.962305    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:50:52.731620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.731586    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"]
Jul 17 00:50:52.732016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.731945    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:50:52.732016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.731957    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:50:52.732016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.731980    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:50:52.732016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.731986    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:50:52.732160 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.732040    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:50:52.732160 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.732047    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:50:52.734863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.734847    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"
Jul 17 00:50:52.741004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.740941    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"]
Jul 17 00:50:52.855138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.855109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3401363-fd50-429a-a912-89f7cd2ca646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh\" (UID: \"f3401363-fd50-429a-a912-89f7cd2ca646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"
Jul 17 00:50:52.956049 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.956017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3401363-fd50-429a-a912-89f7cd2ca646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh\" (UID: \"f3401363-fd50-429a-a912-89f7cd2ca646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"
Jul 17 00:50:52.956609 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:52.956588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3401363-fd50-429a-a912-89f7cd2ca646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh\" (UID: \"f3401363-fd50-429a-a912-89f7cd2ca646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"
Jul 17 00:50:53.046393 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:53.046345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"
Jul 17 00:50:53.183903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:53.183874    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"]
Jul 17 00:50:53.185768 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:50:53.185734    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3401363_fd50_429a_a912_89f7cd2ca646.slice/crio-ea018d7271c90a101ab6a2cf26cb377f34af7640dafc5119541f093d46bccf7a WatchSource:0}: Error finding container ea018d7271c90a101ab6a2cf26cb377f34af7640dafc5119541f093d46bccf7a: Status 404 returned error can't find the container with id ea018d7271c90a101ab6a2cf26cb377f34af7640dafc5119541f093d46bccf7a
Jul 17 00:50:53.851942 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:53.851910    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3401363-fd50-429a-a912-89f7cd2ca646" containerID="ccbb881b52acee172c30b457a6446b40d56f320a1de41048bdffce4770228bb5" exitCode=6
Jul 17 00:50:53.852351 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:53.852007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" event={"ID":"f3401363-fd50-429a-a912-89f7cd2ca646","Type":"ContainerDied","Data":"ccbb881b52acee172c30b457a6446b40d56f320a1de41048bdffce4770228bb5"}
Jul 17 00:50:53.852351 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:53.852070    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" event={"ID":"f3401363-fd50-429a-a912-89f7cd2ca646","Type":"ContainerStarted","Data":"ea018d7271c90a101ab6a2cf26cb377f34af7640dafc5119541f093d46bccf7a"}
Jul 17 00:50:53.852351 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:53.852217    2573 scope.go:117] "RemoveContainer" containerID="ccbb881b52acee172c30b457a6446b40d56f320a1de41048bdffce4770228bb5"
Jul 17 00:50:53.962317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:53.962285    2573 scope.go:117] "RemoveContainer" containerID="ea79f4ae31b40a385646388f86b30fbd66b39d06f521aa7a0f52a18bd476d2ad"
Jul 17 00:50:54.860395 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.860358    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3401363-fd50-429a-a912-89f7cd2ca646" containerID="ad3bfd3256963428636eb11b2e823f9a9463f684adf119176dd24ead2e5b2cfe" exitCode=6
Jul 17 00:50:54.860893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.860437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" event={"ID":"f3401363-fd50-429a-a912-89f7cd2ca646","Type":"ContainerDied","Data":"ad3bfd3256963428636eb11b2e823f9a9463f684adf119176dd24ead2e5b2cfe"}
Jul 17 00:50:54.860893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.860534    2573 scope.go:117] "RemoveContainer" containerID="ccbb881b52acee172c30b457a6446b40d56f320a1de41048bdffce4770228bb5"
Jul 17 00:50:54.860893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.860779    2573 scope.go:117] "RemoveContainer" containerID="ad3bfd3256963428636eb11b2e823f9a9463f684adf119176dd24ead2e5b2cfe"
Jul 17 00:50:54.861099 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:54.861074    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh_odh-ai-gateway-infra(f3401363-fd50-429a-a912-89f7cd2ca646)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" podUID="f3401363-fd50-429a-a912-89f7cd2ca646"
Jul 17 00:50:54.862622 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.862599    2573 generic.go:356] "Generic (PLEG): container finished" podID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerID="dc29c6b589ad058ba8894d0fc794f8d18cd05bf3a35796a44c5dc0f12469b91d" exitCode=6
Jul 17 00:50:54.862729 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.862668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" event={"ID":"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb","Type":"ContainerDied","Data":"dc29c6b589ad058ba8894d0fc794f8d18cd05bf3a35796a44c5dc0f12469b91d"}
Jul 17 00:50:54.863015 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.863001    2573 scope.go:117] "RemoveContainer" containerID="dc29c6b589ad058ba8894d0fc794f8d18cd05bf3a35796a44c5dc0f12469b91d"
Jul 17 00:50:54.863237 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:54.863179    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6987b_odh-ai-gateway-infra(8444deb2-dfb7-4c93-92f6-aa55a4ff54cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb"
Jul 17 00:50:54.872748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:54.872719    2573 scope.go:117] "RemoveContainer" containerID="ea79f4ae31b40a385646388f86b30fbd66b39d06f521aa7a0f52a18bd476d2ad"
Jul 17 00:50:55.870410 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:55.870376    2573 scope.go:117] "RemoveContainer" containerID="ad3bfd3256963428636eb11b2e823f9a9463f684adf119176dd24ead2e5b2cfe"
Jul 17 00:50:55.870879 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:50:55.870633    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh_odh-ai-gateway-infra(f3401363-fd50-429a-a912-89f7cd2ca646)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" podUID="f3401363-fd50-429a-a912-89f7cd2ca646"
Jul 17 00:50:55.896316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:55.896275    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"]
Jul 17 00:50:56.040132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.040101    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"
Jul 17 00:50:56.183720 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.183691    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8444deb2-dfb7-4c93-92f6-aa55a4ff54cb-maas-api-service-ca\") pod \"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb\" (UID: \"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb\") "
Jul 17 00:50:56.184101 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.184079    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8444deb2-dfb7-4c93-92f6-aa55a4ff54cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" (UID: "8444deb2-dfb7-4c93-92f6-aa55a4ff54cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:50:56.284958 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.284914    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8444deb2-dfb7-4c93-92f6-aa55a4ff54cb-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:50:56.878094 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.878062    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"
Jul 17 00:50:56.878578 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.878091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b" event={"ID":"8444deb2-dfb7-4c93-92f6-aa55a4ff54cb","Type":"ContainerDied","Data":"8ff55a91ba3ad7ea2315ef2086db106fbe2267d819c5cfcdca327239508e6183"}
Jul 17 00:50:56.878578 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.878167    2573 scope.go:117] "RemoveContainer" containerID="dc29c6b589ad058ba8894d0fc794f8d18cd05bf3a35796a44c5dc0f12469b91d"
Jul 17 00:50:56.898521 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.898486    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"]
Jul 17 00:50:56.901435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:56.901401    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6987b"]
Jul 17 00:50:57.968293 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:50:57.968257    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" path="/var/lib/kubelet/pods/8444deb2-dfb7-4c93-92f6-aa55a4ff54cb/volumes"
Jul 17 00:51:01.962386 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:01.962355    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:51:01.962906 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:01.962521    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:51:10.963131 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:10.963053    2573 scope.go:117] "RemoveContainer" containerID="ad3bfd3256963428636eb11b2e823f9a9463f684adf119176dd24ead2e5b2cfe"
Jul 17 00:51:11.967026 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:11.966928    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3401363-fd50-429a-a912-89f7cd2ca646" containerID="ced490ccc7aaf1109dd7bc62c18a26f01f44e998783798d0c0fa96fc30e67af8" exitCode=6
Jul 17 00:51:11.969003 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:11.968976    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" event={"ID":"f3401363-fd50-429a-a912-89f7cd2ca646","Type":"ContainerDied","Data":"ced490ccc7aaf1109dd7bc62c18a26f01f44e998783798d0c0fa96fc30e67af8"}
Jul 17 00:51:11.969088 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:11.969051    2573 scope.go:117] "RemoveContainer" containerID="ad3bfd3256963428636eb11b2e823f9a9463f684adf119176dd24ead2e5b2cfe"
Jul 17 00:51:11.969469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:11.969444    2573 scope.go:117] "RemoveContainer" containerID="ced490ccc7aaf1109dd7bc62c18a26f01f44e998783798d0c0fa96fc30e67af8"
Jul 17 00:51:11.969726 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:11.969700    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh_odh-ai-gateway-infra(f3401363-fd50-429a-a912-89f7cd2ca646)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" podUID="f3401363-fd50-429a-a912-89f7cd2ca646"
Jul 17 00:51:12.993061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:12.993021    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"]
Jul 17 00:51:13.137012 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.136987    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"
Jul 17 00:51:13.248919 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.248822    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3401363-fd50-429a-a912-89f7cd2ca646-maas-api-service-ca\") pod \"f3401363-fd50-429a-a912-89f7cd2ca646\" (UID: \"f3401363-fd50-429a-a912-89f7cd2ca646\") "
Jul 17 00:51:13.249195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.249170    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3401363-fd50-429a-a912-89f7cd2ca646-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3401363-fd50-429a-a912-89f7cd2ca646" (UID: "f3401363-fd50-429a-a912-89f7cd2ca646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:51:13.349817 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.349752    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3401363-fd50-429a-a912-89f7cd2ca646-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:51:13.983301 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.983278    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"
Jul 17 00:51:13.983456 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.983275    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh" event={"ID":"f3401363-fd50-429a-a912-89f7cd2ca646","Type":"ContainerDied","Data":"ea018d7271c90a101ab6a2cf26cb377f34af7640dafc5119541f093d46bccf7a"}
Jul 17 00:51:13.983456 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.983408    2573 scope.go:117] "RemoveContainer" containerID="ced490ccc7aaf1109dd7bc62c18a26f01f44e998783798d0c0fa96fc30e67af8"
Jul 17 00:51:13.998087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:13.998062    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"]
Jul 17 00:51:14.004614 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:14.004590    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-gwgmh"]
Jul 17 00:51:15.965462 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:15.965433    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:51:15.965953 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:15.965644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:51:15.968216 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:15.968196    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" path="/var/lib/kubelet/pods/f3401363-fd50-429a-a912-89f7cd2ca646/volumes"
Jul 17 00:51:18.480507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480471    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"]
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480847    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480859    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480867    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480873    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480884    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480889    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480900    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480908    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480916    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480921    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480976    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480984    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480991    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f7c4dc-745d-4475-ba7d-2f2ef3f875a4" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.480996    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.481003    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:18.481005 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.481009    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:18.485150 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.485131    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"
Jul 17 00:51:18.487175 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.487142    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:51:18.489554 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.489529    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"]
Jul 17 00:51:18.497374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.497348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8738d7-0c9a-43c2-9fb5-bbae52747400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2\" (UID: \"0c8738d7-0c9a-43c2-9fb5-bbae52747400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"
Jul 17 00:51:18.598152 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.598109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8738d7-0c9a-43c2-9fb5-bbae52747400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2\" (UID: \"0c8738d7-0c9a-43c2-9fb5-bbae52747400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"
Jul 17 00:51:18.598745 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.598722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8738d7-0c9a-43c2-9fb5-bbae52747400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2\" (UID: \"0c8738d7-0c9a-43c2-9fb5-bbae52747400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"
Jul 17 00:51:18.798509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.798473    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"
Jul 17 00:51:18.928045 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:18.928014    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"]
Jul 17 00:51:18.929806 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:51:18.929752    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8738d7_0c9a_43c2_9fb5_bbae52747400.slice/crio-7e2aabf12daaf27f73941c90f7250d57018465abd424ff7e8c53305c5dedcb03 WatchSource:0}: Error finding container 7e2aabf12daaf27f73941c90f7250d57018465abd424ff7e8c53305c5dedcb03: Status 404 returned error can't find the container with id 7e2aabf12daaf27f73941c90f7250d57018465abd424ff7e8c53305c5dedcb03
Jul 17 00:51:19.010846 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:19.010779    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" event={"ID":"0c8738d7-0c9a-43c2-9fb5-bbae52747400","Type":"ContainerStarted","Data":"7e2aabf12daaf27f73941c90f7250d57018465abd424ff7e8c53305c5dedcb03"}
Jul 17 00:51:20.017499 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:20.017467    2573 generic.go:356] "Generic (PLEG): container finished" podID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerID="0a5917e410c6e76bf58c7e38fbd11e6148866e7bf79ba172db4c116b35b5429e" exitCode=7
Jul 17 00:51:20.017957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:20.017558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" event={"ID":"0c8738d7-0c9a-43c2-9fb5-bbae52747400","Type":"ContainerDied","Data":"0a5917e410c6e76bf58c7e38fbd11e6148866e7bf79ba172db4c116b35b5429e"}
Jul 17 00:51:20.017957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:20.017733    2573 scope.go:117] "RemoveContainer" containerID="0a5917e410c6e76bf58c7e38fbd11e6148866e7bf79ba172db4c116b35b5429e"
Jul 17 00:51:21.024442 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:21.024407    2573 generic.go:356] "Generic (PLEG): container finished" podID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerID="eef5eeb51865e9296fa561a84a32446caa212efcebc03d3126a2b4e27645ab00" exitCode=7
Jul 17 00:51:21.024841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:21.024483    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" event={"ID":"0c8738d7-0c9a-43c2-9fb5-bbae52747400","Type":"ContainerDied","Data":"eef5eeb51865e9296fa561a84a32446caa212efcebc03d3126a2b4e27645ab00"}
Jul 17 00:51:21.024841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:21.024538    2573 scope.go:117] "RemoveContainer" containerID="0a5917e410c6e76bf58c7e38fbd11e6148866e7bf79ba172db4c116b35b5429e"
Jul 17 00:51:21.024841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:21.024760    2573 scope.go:117] "RemoveContainer" containerID="eef5eeb51865e9296fa561a84a32446caa212efcebc03d3126a2b4e27645ab00"
Jul 17 00:51:21.025005 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:21.024984    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2_odh-ai-gateway-infra(0c8738d7-0c9a-43c2-9fb5-bbae52747400)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400"
Jul 17 00:51:22.031624 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:22.031598    2573 scope.go:117] "RemoveContainer" containerID="eef5eeb51865e9296fa561a84a32446caa212efcebc03d3126a2b4e27645ab00"
Jul 17 00:51:22.032127 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:22.031826    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2_odh-ai-gateway-infra(0c8738d7-0c9a-43c2-9fb5-bbae52747400)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400"
Jul 17 00:51:30.318017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.317985    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"]
Jul 17 00:51:30.318512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.318353    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:30.318512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.318363    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:30.318512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.318372    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:30.318512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.318378    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3401363-fd50-429a-a912-89f7cd2ca646" containerName="revoke-keys"
Jul 17 00:51:30.318512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.318450    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8444deb2-dfb7-4c93-92f6-aa55a4ff54cb" containerName="revoke-keys"
Jul 17 00:51:30.322617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.322599    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"
Jul 17 00:51:30.327098 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.327061    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"]
Jul 17 00:51:30.401201 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.401175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36492e6-ac7e-49a6-85eb-e09da1c64d59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z\" (UID: \"b36492e6-ac7e-49a6-85eb-e09da1c64d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"
Jul 17 00:51:30.502451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.502428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36492e6-ac7e-49a6-85eb-e09da1c64d59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z\" (UID: \"b36492e6-ac7e-49a6-85eb-e09da1c64d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"
Jul 17 00:51:30.503113 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.503091    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36492e6-ac7e-49a6-85eb-e09da1c64d59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z\" (UID: \"b36492e6-ac7e-49a6-85eb-e09da1c64d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"
Jul 17 00:51:30.635941 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.635875    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"
Jul 17 00:51:30.766282 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.766254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"]
Jul 17 00:51:30.766650 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:51:30.766629    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36492e6_ac7e_49a6_85eb_e09da1c64d59.slice/crio-e1df3fe04f1701b5c0023d180f37d85c68e7588d25bdada3c2dde573e92697a9 WatchSource:0}: Error finding container e1df3fe04f1701b5c0023d180f37d85c68e7588d25bdada3c2dde573e92697a9: Status 404 returned error can't find the container with id e1df3fe04f1701b5c0023d180f37d85c68e7588d25bdada3c2dde573e92697a9
Jul 17 00:51:30.962778 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:30.962756    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:51:30.962964 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:30.962944    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:51:31.078741 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:31.078711    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" event={"ID":"b36492e6-ac7e-49a6-85eb-e09da1c64d59","Type":"ContainerStarted","Data":"37b7e716802f8f02192f06a9f5f32667f60da9e2f46ed121c2b9f530de180944"}
Jul 17 00:51:31.078870 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:31.078767    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" event={"ID":"b36492e6-ac7e-49a6-85eb-e09da1c64d59","Type":"ContainerStarted","Data":"e1df3fe04f1701b5c0023d180f37d85c68e7588d25bdada3c2dde573e92697a9"}
Jul 17 00:51:31.094929 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:31.094885    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" podStartSLOduration=1.094872043 podStartE2EDuration="1.094872043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:51:31.091963038 +0000 UTC m=+2085.668115606" watchObservedRunningTime="2026-07-17 00:51:31.094872043 +0000 UTC m=+2085.671024616"
Jul 17 00:51:32.085311 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:32.085275    2573 generic.go:356] "Generic (PLEG): container finished" podID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerID="37b7e716802f8f02192f06a9f5f32667f60da9e2f46ed121c2b9f530de180944" exitCode=6
Jul 17 00:51:32.085693 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:32.085337    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" event={"ID":"b36492e6-ac7e-49a6-85eb-e09da1c64d59","Type":"ContainerDied","Data":"37b7e716802f8f02192f06a9f5f32667f60da9e2f46ed121c2b9f530de180944"}
Jul 17 00:51:32.085693 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:32.085479    2573 scope.go:117] "RemoveContainer" containerID="37b7e716802f8f02192f06a9f5f32667f60da9e2f46ed121c2b9f530de180944"
Jul 17 00:51:33.092381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:33.092355    2573 generic.go:356] "Generic (PLEG): container finished" podID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerID="424168ae24064cf11a2b98aca095d2677ec5530d9604c70789d50a95f311e459" exitCode=6
Jul 17 00:51:33.092738 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:33.092444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" event={"ID":"b36492e6-ac7e-49a6-85eb-e09da1c64d59","Type":"ContainerDied","Data":"424168ae24064cf11a2b98aca095d2677ec5530d9604c70789d50a95f311e459"}
Jul 17 00:51:33.092738 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:33.092499    2573 scope.go:117] "RemoveContainer" containerID="37b7e716802f8f02192f06a9f5f32667f60da9e2f46ed121c2b9f530de180944"
Jul 17 00:51:33.092738 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:33.092593    2573 scope.go:117] "RemoveContainer" containerID="424168ae24064cf11a2b98aca095d2677ec5530d9604c70789d50a95f311e459"
Jul 17 00:51:33.092933 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:33.092841    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z_odh-ai-gateway-infra(b36492e6-ac7e-49a6-85eb-e09da1c64d59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59"
Jul 17 00:51:34.099039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:34.099010    2573 scope.go:117] "RemoveContainer" containerID="424168ae24064cf11a2b98aca095d2677ec5530d9604c70789d50a95f311e459"
Jul 17 00:51:34.099488 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:34.099235    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z_odh-ai-gateway-infra(b36492e6-ac7e-49a6-85eb-e09da1c64d59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59"
Jul 17 00:51:35.965577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:35.965547    2573 scope.go:117] "RemoveContainer" containerID="eef5eeb51865e9296fa561a84a32446caa212efcebc03d3126a2b4e27645ab00"
Jul 17 00:51:37.116296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:37.116257    2573 generic.go:356] "Generic (PLEG): container finished" podID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerID="74794f4fa9814e5d9c3579bf626219ceb9a04f1d6b8aa658b37285f9cc33cf2b" exitCode=7
Jul 17 00:51:37.116704 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:37.116328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" event={"ID":"0c8738d7-0c9a-43c2-9fb5-bbae52747400","Type":"ContainerDied","Data":"74794f4fa9814e5d9c3579bf626219ceb9a04f1d6b8aa658b37285f9cc33cf2b"}
Jul 17 00:51:37.116704 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:37.116412    2573 scope.go:117] "RemoveContainer" containerID="eef5eeb51865e9296fa561a84a32446caa212efcebc03d3126a2b4e27645ab00"
Jul 17 00:51:37.116811 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:37.116712    2573 scope.go:117] "RemoveContainer" containerID="74794f4fa9814e5d9c3579bf626219ceb9a04f1d6b8aa658b37285f9cc33cf2b"
Jul 17 00:51:37.116938 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:37.116914    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2_odh-ai-gateway-infra(0c8738d7-0c9a-43c2-9fb5-bbae52747400)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400"
Jul 17 00:51:38.141522 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:38.141480    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"]
Jul 17 00:51:38.300213 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:38.300191    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"
Jul 17 00:51:38.372546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:38.372520    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8738d7-0c9a-43c2-9fb5-bbae52747400-maas-api-service-ca\") pod \"0c8738d7-0c9a-43c2-9fb5-bbae52747400\" (UID: \"0c8738d7-0c9a-43c2-9fb5-bbae52747400\") "
Jul 17 00:51:38.372916 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:38.372888    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8738d7-0c9a-43c2-9fb5-bbae52747400-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c8738d7-0c9a-43c2-9fb5-bbae52747400" (UID: "0c8738d7-0c9a-43c2-9fb5-bbae52747400"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:51:38.473504 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:38.473453    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8738d7-0c9a-43c2-9fb5-bbae52747400-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:51:39.129676 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:39.129643    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"
Jul 17 00:51:39.129909 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:39.129689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2" event={"ID":"0c8738d7-0c9a-43c2-9fb5-bbae52747400","Type":"ContainerDied","Data":"7e2aabf12daaf27f73941c90f7250d57018465abd424ff7e8c53305c5dedcb03"}
Jul 17 00:51:39.129909 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:39.129752    2573 scope.go:117] "RemoveContainer" containerID="74794f4fa9814e5d9c3579bf626219ceb9a04f1d6b8aa658b37285f9cc33cf2b"
Jul 17 00:51:39.150099 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:39.150074    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"]
Jul 17 00:51:39.155602 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:39.155578    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-qvsj2"]
Jul 17 00:51:39.967960 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:39.967930    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" path="/var/lib/kubelet/pods/0c8738d7-0c9a-43c2-9fb5-bbae52747400/volumes"
Jul 17 00:51:44.962598 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:44.962566    2573 scope.go:117] "RemoveContainer" containerID="424168ae24064cf11a2b98aca095d2677ec5530d9604c70789d50a95f311e459"
Jul 17 00:51:45.965138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:45.965076    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:51:45.965571 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:45.965243    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:51:46.169684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:46.169654    2573 generic.go:356] "Generic (PLEG): container finished" podID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerID="5663ecb0ead8819f5bd899ad851ac98bb1679c87fa06a19fcb0c524dcab14afd" exitCode=6
Jul 17 00:51:46.169828 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:46.169695    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" event={"ID":"b36492e6-ac7e-49a6-85eb-e09da1c64d59","Type":"ContainerDied","Data":"5663ecb0ead8819f5bd899ad851ac98bb1679c87fa06a19fcb0c524dcab14afd"}
Jul 17 00:51:46.169828 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:46.169745    2573 scope.go:117] "RemoveContainer" containerID="424168ae24064cf11a2b98aca095d2677ec5530d9604c70789d50a95f311e459"
Jul 17 00:51:46.170046 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:46.170019    2573 scope.go:117] "RemoveContainer" containerID="5663ecb0ead8819f5bd899ad851ac98bb1679c87fa06a19fcb0c524dcab14afd"
Jul 17 00:51:46.170238 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:46.170216    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z_odh-ai-gateway-infra(b36492e6-ac7e-49a6-85eb-e09da1c64d59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59"
Jul 17 00:51:47.192832 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:47.192785    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"]
Jul 17 00:51:47.340478 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:47.340443    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"
Jul 17 00:51:47.443705 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:47.443604    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36492e6-ac7e-49a6-85eb-e09da1c64d59-maas-api-service-ca\") pod \"b36492e6-ac7e-49a6-85eb-e09da1c64d59\" (UID: \"b36492e6-ac7e-49a6-85eb-e09da1c64d59\") "
Jul 17 00:51:47.443999 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:47.443975    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36492e6-ac7e-49a6-85eb-e09da1c64d59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b36492e6-ac7e-49a6-85eb-e09da1c64d59" (UID: "b36492e6-ac7e-49a6-85eb-e09da1c64d59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:51:47.544631 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:47.544589    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36492e6-ac7e-49a6-85eb-e09da1c64d59-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:51:47.829621 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:47.829516    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:51:47.849359 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:47.837979    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:51:48.184436 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:48.184346    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"
Jul 17 00:51:48.184633 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:48.184436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z" event={"ID":"b36492e6-ac7e-49a6-85eb-e09da1c64d59","Type":"ContainerDied","Data":"e1df3fe04f1701b5c0023d180f37d85c68e7588d25bdada3c2dde573e92697a9"}
Jul 17 00:51:48.184633 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:48.184498    2573 scope.go:117] "RemoveContainer" containerID="5663ecb0ead8819f5bd899ad851ac98bb1679c87fa06a19fcb0c524dcab14afd"
Jul 17 00:51:48.203509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:48.203475    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"]
Jul 17 00:51:48.208887 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:48.208849    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-g7p8z"]
Jul 17 00:51:49.968115 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:49.968079    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" path="/var/lib/kubelet/pods/b36492e6-ac7e-49a6-85eb-e09da1c64d59/volumes"
Jul 17 00:51:52.779217 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779186    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"]
Jul 17 00:51:52.779577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779533    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779543    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779553    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:51:52.779577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779559    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:51:52.779577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779567    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779574    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779582    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779588    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779593    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779599    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779668    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779677    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779684    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779693    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.779772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.779698    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8738d7-0c9a-43c2-9fb5-bbae52747400" containerName="revoke-keys"
Jul 17 00:51:52.782798 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.782771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"
Jul 17 00:51:52.785442 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.785422    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:51:52.791680 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.791655    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"]
Jul 17 00:51:52.888689 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.888666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4be2619-016e-4b24-96b4-94eed53bbc51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl\" (UID: \"e4be2619-016e-4b24-96b4-94eed53bbc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"
Jul 17 00:51:52.989864 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.989839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4be2619-016e-4b24-96b4-94eed53bbc51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl\" (UID: \"e4be2619-016e-4b24-96b4-94eed53bbc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"
Jul 17 00:51:52.990443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:52.990421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4be2619-016e-4b24-96b4-94eed53bbc51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl\" (UID: \"e4be2619-016e-4b24-96b4-94eed53bbc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"
Jul 17 00:51:53.094081 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:53.094030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"
Jul 17 00:51:53.220603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:53.220574    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"]
Jul 17 00:51:53.222448 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:51:53.222422    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4be2619_016e_4b24_96b4_94eed53bbc51.slice/crio-4c1def862a0457694aa31804695d5a3a5c91612e8dfda3046a792fa27c39488d WatchSource:0}: Error finding container 4c1def862a0457694aa31804695d5a3a5c91612e8dfda3046a792fa27c39488d: Status 404 returned error can't find the container with id 4c1def862a0457694aa31804695d5a3a5c91612e8dfda3046a792fa27c39488d
Jul 17 00:51:54.221329 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:54.221235    2573 generic.go:356] "Generic (PLEG): container finished" podID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerID="bb13d033b2267cdf3d673570d81471da6413ca64825a12db5c436852dd3c10af" exitCode=6
Jul 17 00:51:54.221329 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:54.221293    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" event={"ID":"e4be2619-016e-4b24-96b4-94eed53bbc51","Type":"ContainerDied","Data":"bb13d033b2267cdf3d673570d81471da6413ca64825a12db5c436852dd3c10af"}
Jul 17 00:51:54.221721 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:54.221333    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" event={"ID":"e4be2619-016e-4b24-96b4-94eed53bbc51","Type":"ContainerStarted","Data":"4c1def862a0457694aa31804695d5a3a5c91612e8dfda3046a792fa27c39488d"}
Jul 17 00:51:54.221721 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:54.221475    2573 scope.go:117] "RemoveContainer" containerID="bb13d033b2267cdf3d673570d81471da6413ca64825a12db5c436852dd3c10af"
Jul 17 00:51:55.229073 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:55.228994    2573 generic.go:356] "Generic (PLEG): container finished" podID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerID="9a4d66f818cf2f1889d8d4a2baf3ab111ca0651e5601a8b947d1b74c08b5f741" exitCode=6
Jul 17 00:51:55.229431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:55.229076    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" event={"ID":"e4be2619-016e-4b24-96b4-94eed53bbc51","Type":"ContainerDied","Data":"9a4d66f818cf2f1889d8d4a2baf3ab111ca0651e5601a8b947d1b74c08b5f741"}
Jul 17 00:51:55.229431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:55.229140    2573 scope.go:117] "RemoveContainer" containerID="bb13d033b2267cdf3d673570d81471da6413ca64825a12db5c436852dd3c10af"
Jul 17 00:51:55.229431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:55.229365    2573 scope.go:117] "RemoveContainer" containerID="9a4d66f818cf2f1889d8d4a2baf3ab111ca0651e5601a8b947d1b74c08b5f741"
Jul 17 00:51:55.229577 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:55.229559    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl_odh-ai-gateway-infra(e4be2619-016e-4b24-96b4-94eed53bbc51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51"
Jul 17 00:51:56.235721 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:56.235696    2573 scope.go:117] "RemoveContainer" containerID="9a4d66f818cf2f1889d8d4a2baf3ab111ca0651e5601a8b947d1b74c08b5f741"
Jul 17 00:51:56.236072 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:56.235890    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl_odh-ai-gateway-infra(e4be2619-016e-4b24-96b4-94eed53bbc51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51"
Jul 17 00:51:59.962868 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:51:59.962836    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:51:59.963250 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:51:59.963015    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:52:09.963074 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:09.963040    2573 scope.go:117] "RemoveContainer" containerID="9a4d66f818cf2f1889d8d4a2baf3ab111ca0651e5601a8b947d1b74c08b5f741"
Jul 17 00:52:10.306061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:10.306034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" event={"ID":"e4be2619-016e-4b24-96b4-94eed53bbc51","Type":"ContainerStarted","Data":"861fe5a9e28d00e2e91db59c9bf0d63454bc1ce6028ca745dc7c4af942919bdf"}
Jul 17 00:52:10.320000 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:10.319960    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" podStartSLOduration=18.319945138 podStartE2EDuration="18.319945138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:52:10.31890523 +0000 UTC m=+2124.895057805" watchObservedRunningTime="2026-07-17 00:52:10.319945138 +0000 UTC m=+2124.896097767"
Jul 17 00:52:10.962962 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:10.962884    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:52:10.963110 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:10.963041    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:52:11.313191 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:11.313163    2573 generic.go:356] "Generic (PLEG): container finished" podID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerID="861fe5a9e28d00e2e91db59c9bf0d63454bc1ce6028ca745dc7c4af942919bdf" exitCode=6
Jul 17 00:52:11.313347 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:11.313235    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" event={"ID":"e4be2619-016e-4b24-96b4-94eed53bbc51","Type":"ContainerDied","Data":"861fe5a9e28d00e2e91db59c9bf0d63454bc1ce6028ca745dc7c4af942919bdf"}
Jul 17 00:52:11.313347 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:11.313294    2573 scope.go:117] "RemoveContainer" containerID="9a4d66f818cf2f1889d8d4a2baf3ab111ca0651e5601a8b947d1b74c08b5f741"
Jul 17 00:52:11.313551 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:11.313537    2573 scope.go:117] "RemoveContainer" containerID="861fe5a9e28d00e2e91db59c9bf0d63454bc1ce6028ca745dc7c4af942919bdf"
Jul 17 00:52:11.313746 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:11.313727    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl_odh-ai-gateway-infra(e4be2619-016e-4b24-96b4-94eed53bbc51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51"
Jul 17 00:52:11.328461 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:11.328440    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"]
Jul 17 00:52:12.450430 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:12.450405    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"
Jul 17 00:52:12.551342 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:12.551317    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4be2619-016e-4b24-96b4-94eed53bbc51-maas-api-service-ca\") pod \"e4be2619-016e-4b24-96b4-94eed53bbc51\" (UID: \"e4be2619-016e-4b24-96b4-94eed53bbc51\") "
Jul 17 00:52:12.551627 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:12.551605    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4be2619-016e-4b24-96b4-94eed53bbc51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4be2619-016e-4b24-96b4-94eed53bbc51" (UID: "e4be2619-016e-4b24-96b4-94eed53bbc51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:52:12.652160 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:12.652108    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4be2619-016e-4b24-96b4-94eed53bbc51-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:52:13.326452 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:13.326422    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"
Jul 17 00:52:13.326650 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:13.326460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl" event={"ID":"e4be2619-016e-4b24-96b4-94eed53bbc51","Type":"ContainerDied","Data":"4c1def862a0457694aa31804695d5a3a5c91612e8dfda3046a792fa27c39488d"}
Jul 17 00:52:13.326650 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:13.326539    2573 scope.go:117] "RemoveContainer" containerID="861fe5a9e28d00e2e91db59c9bf0d63454bc1ce6028ca745dc7c4af942919bdf"
Jul 17 00:52:13.347110 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:13.347084    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"]
Jul 17 00:52:13.350744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:13.350723    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hv4kl"]
Jul 17 00:52:13.967706 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:13.967675    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" path="/var/lib/kubelet/pods/e4be2619-016e-4b24-96b4-94eed53bbc51/volumes"
Jul 17 00:52:18.519722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.519688    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"]
Jul 17 00:52:18.520120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520054    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:18.520120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520065    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:18.520120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520075    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:52:18.520120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520081    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:52:18.520120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520104    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:18.520120 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520110    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:18.520314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520168    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:18.520314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520176    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:18.520314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520182    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36492e6-ac7e-49a6-85eb-e09da1c64d59" containerName="revoke-keys"
Jul 17 00:52:18.520314 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.520189    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:18.524378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.524358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"
Jul 17 00:52:18.526364 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.526338    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:52:18.532283 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.532100    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"]
Jul 17 00:52:18.601061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.601020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97da358a-2be9-4623-8c66-c477c7248d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6\" (UID: \"97da358a-2be9-4623-8c66-c477c7248d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"
Jul 17 00:52:18.702139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.702105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97da358a-2be9-4623-8c66-c477c7248d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6\" (UID: \"97da358a-2be9-4623-8c66-c477c7248d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"
Jul 17 00:52:18.702772 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.702745    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97da358a-2be9-4623-8c66-c477c7248d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6\" (UID: \"97da358a-2be9-4623-8c66-c477c7248d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"
Jul 17 00:52:18.836016 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.835939    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"
Jul 17 00:52:18.966600 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:18.966576    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"]
Jul 17 00:52:18.968896 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:52:18.968860    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97da358a_2be9_4623_8c66_c477c7248d37.slice/crio-3e8e655cf40744351ae07af30dfa2de898b56ce2faa12d300ce7195b59d89372 WatchSource:0}: Error finding container 3e8e655cf40744351ae07af30dfa2de898b56ce2faa12d300ce7195b59d89372: Status 404 returned error can't find the container with id 3e8e655cf40744351ae07af30dfa2de898b56ce2faa12d300ce7195b59d89372
Jul 17 00:52:19.360094 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:19.360058    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" event={"ID":"97da358a-2be9-4623-8c66-c477c7248d37","Type":"ContainerStarted","Data":"9a5acaa446a96366fadf70004c86b8bd52b0830e35747debecb08812a4d1fab6"}
Jul 17 00:52:19.360252 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:19.360108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" event={"ID":"97da358a-2be9-4623-8c66-c477c7248d37","Type":"ContainerStarted","Data":"3e8e655cf40744351ae07af30dfa2de898b56ce2faa12d300ce7195b59d89372"}
Jul 17 00:52:19.374836 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:19.374772    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" podStartSLOduration=1.374737657 podStartE2EDuration="1.374737657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:52:19.373012505 +0000 UTC m=+2133.949165078" watchObservedRunningTime="2026-07-17 00:52:19.374737657 +0000 UTC m=+2133.950890231"
Jul 17 00:52:20.356568 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.356534    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"]
Jul 17 00:52:20.356956 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.356928    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:20.356956 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.356944    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4be2619-016e-4b24-96b4-94eed53bbc51" containerName="revoke-keys"
Jul 17 00:52:20.359774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.359758    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"
Jul 17 00:52:20.368719 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.368694    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"]
Jul 17 00:52:20.369380 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.369357    2573 generic.go:356] "Generic (PLEG): container finished" podID="97da358a-2be9-4623-8c66-c477c7248d37" containerID="9a5acaa446a96366fadf70004c86b8bd52b0830e35747debecb08812a4d1fab6" exitCode=7
Jul 17 00:52:20.369587 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.369438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" event={"ID":"97da358a-2be9-4623-8c66-c477c7248d37","Type":"ContainerDied","Data":"9a5acaa446a96366fadf70004c86b8bd52b0830e35747debecb08812a4d1fab6"}
Jul 17 00:52:20.369757 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.369718    2573 scope.go:117] "RemoveContainer" containerID="9a5acaa446a96366fadf70004c86b8bd52b0830e35747debecb08812a4d1fab6"
Jul 17 00:52:20.515284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.515256    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b626b3-acb6-4915-9f12-c8bac5c18395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz\" (UID: \"d8b626b3-acb6-4915-9f12-c8bac5c18395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"
Jul 17 00:52:20.616586 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.616516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b626b3-acb6-4915-9f12-c8bac5c18395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz\" (UID: \"d8b626b3-acb6-4915-9f12-c8bac5c18395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"
Jul 17 00:52:20.617161 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.617143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b626b3-acb6-4915-9f12-c8bac5c18395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz\" (UID: \"d8b626b3-acb6-4915-9f12-c8bac5c18395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"
Jul 17 00:52:20.673478 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.673449    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"
Jul 17 00:52:20.801073 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:20.801051    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"]
Jul 17 00:52:21.376819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:21.376719    2573 generic.go:356] "Generic (PLEG): container finished" podID="97da358a-2be9-4623-8c66-c477c7248d37" containerID="dd7d1061cd3d6c68c090222feacafc7a20b474d538e756728ad3241621041ed1" exitCode=7
Jul 17 00:52:21.377204 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:21.376825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" event={"ID":"97da358a-2be9-4623-8c66-c477c7248d37","Type":"ContainerDied","Data":"dd7d1061cd3d6c68c090222feacafc7a20b474d538e756728ad3241621041ed1"}
Jul 17 00:52:21.377204 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:21.376889    2573 scope.go:117] "RemoveContainer" containerID="9a5acaa446a96366fadf70004c86b8bd52b0830e35747debecb08812a4d1fab6"
Jul 17 00:52:21.377204 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:21.377060    2573 scope.go:117] "RemoveContainer" containerID="dd7d1061cd3d6c68c090222feacafc7a20b474d538e756728ad3241621041ed1"
Jul 17 00:52:21.377350 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:21.377271    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6_odh-ai-gateway-infra(97da358a-2be9-4623-8c66-c477c7248d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" podUID="97da358a-2be9-4623-8c66-c477c7248d37"
Jul 17 00:52:21.378443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:21.378420    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" event={"ID":"d8b626b3-acb6-4915-9f12-c8bac5c18395","Type":"ContainerStarted","Data":"4bb2fa2a16bb84053ed8a02a92ec0acd9f25e54975f15f9468e29adc3973255b"}
Jul 17 00:52:21.378523 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:21.378469    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" event={"ID":"d8b626b3-acb6-4915-9f12-c8bac5c18395","Type":"ContainerStarted","Data":"89a2921152934710ba796c8a3a26cd612207ccde0978ddc09e4564fbc068a106"}
Jul 17 00:52:21.402228 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:21.402190    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" podStartSLOduration=1.40217736 podStartE2EDuration="1.40217736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:52:21.400620847 +0000 UTC m=+2135.976773421" watchObservedRunningTime="2026-07-17 00:52:21.40217736 +0000 UTC m=+2135.978329939"
Jul 17 00:52:22.386265 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:22.386236    2573 scope.go:117] "RemoveContainer" containerID="dd7d1061cd3d6c68c090222feacafc7a20b474d538e756728ad3241621041ed1"
Jul 17 00:52:22.386735 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:22.386463    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6_odh-ai-gateway-infra(97da358a-2be9-4623-8c66-c477c7248d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" podUID="97da358a-2be9-4623-8c66-c477c7248d37"
Jul 17 00:52:22.387559 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:22.387529    2573 generic.go:356] "Generic (PLEG): container finished" podID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerID="4bb2fa2a16bb84053ed8a02a92ec0acd9f25e54975f15f9468e29adc3973255b" exitCode=6
Jul 17 00:52:22.387690 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:22.387571    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" event={"ID":"d8b626b3-acb6-4915-9f12-c8bac5c18395","Type":"ContainerDied","Data":"4bb2fa2a16bb84053ed8a02a92ec0acd9f25e54975f15f9468e29adc3973255b"}
Jul 17 00:52:22.387782 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:22.387765    2573 scope.go:117] "RemoveContainer" containerID="4bb2fa2a16bb84053ed8a02a92ec0acd9f25e54975f15f9468e29adc3973255b"
Jul 17 00:52:23.394424 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:23.394393    2573 generic.go:356] "Generic (PLEG): container finished" podID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerID="b4a355871c89736812d4971765c074ba66393ccb97b9e5d0cd68cd21e1df8713" exitCode=6
Jul 17 00:52:23.394766 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:23.394477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" event={"ID":"d8b626b3-acb6-4915-9f12-c8bac5c18395","Type":"ContainerDied","Data":"b4a355871c89736812d4971765c074ba66393ccb97b9e5d0cd68cd21e1df8713"}
Jul 17 00:52:23.394766 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:23.394543    2573 scope.go:117] "RemoveContainer" containerID="4bb2fa2a16bb84053ed8a02a92ec0acd9f25e54975f15f9468e29adc3973255b"
Jul 17 00:52:23.394883 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:23.394779    2573 scope.go:117] "RemoveContainer" containerID="b4a355871c89736812d4971765c074ba66393ccb97b9e5d0cd68cd21e1df8713"
Jul 17 00:52:23.395005 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:23.394984    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz_odh-ai-gateway-infra(d8b626b3-acb6-4915-9f12-c8bac5c18395)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395"
Jul 17 00:52:23.963382 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:23.963345    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:52:23.963581 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:23.963539    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:52:24.401697 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:24.401672    2573 scope.go:117] "RemoveContainer" containerID="b4a355871c89736812d4971765c074ba66393ccb97b9e5d0cd68cd21e1df8713"
Jul 17 00:52:24.402069 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:24.401849    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz_odh-ai-gateway-infra(d8b626b3-acb6-4915-9f12-c8bac5c18395)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395"
Jul 17 00:52:34.963480 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:34.963449    2573 scope.go:117] "RemoveContainer" containerID="b4a355871c89736812d4971765c074ba66393ccb97b9e5d0cd68cd21e1df8713"
Jul 17 00:52:35.476965 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:35.476936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" event={"ID":"d8b626b3-acb6-4915-9f12-c8bac5c18395","Type":"ContainerStarted","Data":"52eeb41da7e6a5e1870275d9c0e653977e0b4133f8ceb1451dc0967cec0ccdd6"}
Jul 17 00:52:35.967003 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:35.966973    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:52:36.483861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.483825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerStarted","Data":"b8ab4164a1abda6fea0e26bf841885e9e80ca62f41f594303571506a89207745"}
Jul 17 00:52:36.484099 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.484080    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:52:36.485513 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.485489    2573 generic.go:356] "Generic (PLEG): container finished" podID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerID="52eeb41da7e6a5e1870275d9c0e653977e0b4133f8ceb1451dc0967cec0ccdd6" exitCode=6
Jul 17 00:52:36.485618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.485561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" event={"ID":"d8b626b3-acb6-4915-9f12-c8bac5c18395","Type":"ContainerDied","Data":"52eeb41da7e6a5e1870275d9c0e653977e0b4133f8ceb1451dc0967cec0ccdd6"}
Jul 17 00:52:36.485618 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.485610    2573 scope.go:117] "RemoveContainer" containerID="b4a355871c89736812d4971765c074ba66393ccb97b9e5d0cd68cd21e1df8713"
Jul 17 00:52:36.485840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.485819    2573 scope.go:117] "RemoveContainer" containerID="52eeb41da7e6a5e1870275d9c0e653977e0b4133f8ceb1451dc0967cec0ccdd6"
Jul 17 00:52:36.486026 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:36.486008    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz_odh-ai-gateway-infra(d8b626b3-acb6-4915-9f12-c8bac5c18395)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395"
Jul 17 00:52:36.503915 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.503893    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"]
Jul 17 00:52:36.963087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:36.963010    2573 scope.go:117] "RemoveContainer" containerID="dd7d1061cd3d6c68c090222feacafc7a20b474d538e756728ad3241621041ed1"
Jul 17 00:52:37.492540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.492509    2573 generic.go:356] "Generic (PLEG): container finished" podID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerID="b8ab4164a1abda6fea0e26bf841885e9e80ca62f41f594303571506a89207745" exitCode=1
Jul 17 00:52:37.492988 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.492594    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"b8ab4164a1abda6fea0e26bf841885e9e80ca62f41f594303571506a89207745"}
Jul 17 00:52:37.492988 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.492650    2573 scope.go:117] "RemoveContainer" containerID="406aad34a1596d714662bc5078fb182f07843bf60f115e5f15130991bf1d9e51"
Jul 17 00:52:37.492988 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.492775    2573 scope.go:117] "RemoveContainer" containerID="b8ab4164a1abda6fea0e26bf841885e9e80ca62f41f594303571506a89207745"
Jul 17 00:52:37.493124 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:37.492992    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:52:37.497271 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.495246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" event={"ID":"97da358a-2be9-4623-8c66-c477c7248d37","Type":"ContainerStarted","Data":"981b1efe1da0fcdff9e673fe24240b2b97dd738f1286a5b10546d3a860d48b76"}
Jul 17 00:52:37.643556 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.643535    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"
Jul 17 00:52:37.755079 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.755053    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b626b3-acb6-4915-9f12-c8bac5c18395-maas-api-service-ca\") pod \"d8b626b3-acb6-4915-9f12-c8bac5c18395\" (UID: \"d8b626b3-acb6-4915-9f12-c8bac5c18395\") "
Jul 17 00:52:37.755374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.755352    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b626b3-acb6-4915-9f12-c8bac5c18395-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8b626b3-acb6-4915-9f12-c8bac5c18395" (UID: "d8b626b3-acb6-4915-9f12-c8bac5c18395"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:52:37.855959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:37.855907    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b626b3-acb6-4915-9f12-c8bac5c18395-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:52:38.506559 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.506527    2573 scope.go:117] "RemoveContainer" containerID="b8ab4164a1abda6fea0e26bf841885e9e80ca62f41f594303571506a89207745"
Jul 17 00:52:38.507019 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:38.506734    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:52:38.508234 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.508212    2573 generic.go:356] "Generic (PLEG): container finished" podID="97da358a-2be9-4623-8c66-c477c7248d37" containerID="981b1efe1da0fcdff9e673fe24240b2b97dd738f1286a5b10546d3a860d48b76" exitCode=7
Jul 17 00:52:38.508364 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.508275    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" event={"ID":"97da358a-2be9-4623-8c66-c477c7248d37","Type":"ContainerDied","Data":"981b1efe1da0fcdff9e673fe24240b2b97dd738f1286a5b10546d3a860d48b76"}
Jul 17 00:52:38.508364 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.508317    2573 scope.go:117] "RemoveContainer" containerID="dd7d1061cd3d6c68c090222feacafc7a20b474d538e756728ad3241621041ed1"
Jul 17 00:52:38.508586 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.508569    2573 scope.go:117] "RemoveContainer" containerID="981b1efe1da0fcdff9e673fe24240b2b97dd738f1286a5b10546d3a860d48b76"
Jul 17 00:52:38.508751 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:38.508726    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6_odh-ai-gateway-infra(97da358a-2be9-4623-8c66-c477c7248d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" podUID="97da358a-2be9-4623-8c66-c477c7248d37"
Jul 17 00:52:38.509670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.509644    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz" event={"ID":"d8b626b3-acb6-4915-9f12-c8bac5c18395","Type":"ContainerDied","Data":"89a2921152934710ba796c8a3a26cd612207ccde0978ddc09e4564fbc068a106"}
Jul 17 00:52:38.509759 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.509671    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"
Jul 17 00:52:38.520322 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.520306    2573 scope.go:117] "RemoveContainer" containerID="52eeb41da7e6a5e1870275d9c0e653977e0b4133f8ceb1451dc0967cec0ccdd6"
Jul 17 00:52:38.530089 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.530072    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"]
Jul 17 00:52:38.536720 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.536699    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"]
Jul 17 00:52:38.542196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:38.542177    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6vssz"]
Jul 17 00:52:39.654146 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:39.654124    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"
Jul 17 00:52:39.768735 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:39.768660    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97da358a-2be9-4623-8c66-c477c7248d37-maas-api-service-ca\") pod \"97da358a-2be9-4623-8c66-c477c7248d37\" (UID: \"97da358a-2be9-4623-8c66-c477c7248d37\") "
Jul 17 00:52:39.768983 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:39.768963    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97da358a-2be9-4623-8c66-c477c7248d37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97da358a-2be9-4623-8c66-c477c7248d37" (UID: "97da358a-2be9-4623-8c66-c477c7248d37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:52:39.870104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:39.870076    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97da358a-2be9-4623-8c66-c477c7248d37-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:52:39.968513 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:39.968482    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" path="/var/lib/kubelet/pods/d8b626b3-acb6-4915-9f12-c8bac5c18395/volumes"
Jul 17 00:52:40.524644 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:40.524616    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"
Jul 17 00:52:40.524820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:40.524651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6" event={"ID":"97da358a-2be9-4623-8c66-c477c7248d37","Type":"ContainerDied","Data":"3e8e655cf40744351ae07af30dfa2de898b56ce2faa12d300ce7195b59d89372"}
Jul 17 00:52:40.524820 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:40.524728    2573 scope.go:117] "RemoveContainer" containerID="981b1efe1da0fcdff9e673fe24240b2b97dd738f1286a5b10546d3a860d48b76"
Jul 17 00:52:40.543669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:40.543640    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"]
Jul 17 00:52:40.550268 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:40.550246    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-f8hm6"]
Jul 17 00:52:41.967681 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:41.967647    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97da358a-2be9-4623-8c66-c477c7248d37" path="/var/lib/kubelet/pods/97da358a-2be9-4623-8c66-c477c7248d37/volumes"
Jul 17 00:52:44.056354 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:44.056325    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:52:44.056759 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:44.056658    2573 scope.go:117] "RemoveContainer" containerID="b8ab4164a1abda6fea0e26bf841885e9e80ca62f41f594303571506a89207745"
Jul 17 00:52:44.056857 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:44.056836    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8_odh-ai-gateway-infra(62a8b6e4-50ba-4691-933d-5d1c4f8d2003)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003"
Jul 17 00:52:52.826588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.826553    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"]
Jul 17 00:52:52.827165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827147    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:52:52.827220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827169    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:52:52.827220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827184    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827192    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827209    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827218    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827230    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827236    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827246    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827252    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827320    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827327    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827333    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:52:52.827387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.827339    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:52:52.830194 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.830179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"
Jul 17 00:52:52.832154 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.832129    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:52:52.837567 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.837541    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"]
Jul 17 00:52:52.880432 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.880392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3055d1c0-7573-4c25-bbd6-eb94037438f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5\" (UID: \"3055d1c0-7573-4c25-bbd6-eb94037438f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"
Jul 17 00:52:52.981178 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.981143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3055d1c0-7573-4c25-bbd6-eb94037438f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5\" (UID: \"3055d1c0-7573-4c25-bbd6-eb94037438f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"
Jul 17 00:52:52.981718 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:52.981701    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3055d1c0-7573-4c25-bbd6-eb94037438f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5\" (UID: \"3055d1c0-7573-4c25-bbd6-eb94037438f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"
Jul 17 00:52:53.142754 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:53.142659    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"
Jul 17 00:52:53.274620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:53.274596    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"]
Jul 17 00:52:53.276069 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:52:53.276038    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3055d1c0_7573_4c25_bbd6_eb94037438f7.slice/crio-24f1ebbd6d6b3f5405a22bce2c9ad45341972bf79c27df754e31001a980c2be5 WatchSource:0}: Error finding container 24f1ebbd6d6b3f5405a22bce2c9ad45341972bf79c27df754e31001a980c2be5: Status 404 returned error can't find the container with id 24f1ebbd6d6b3f5405a22bce2c9ad45341972bf79c27df754e31001a980c2be5
Jul 17 00:52:53.597391 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:53.597359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" event={"ID":"3055d1c0-7573-4c25-bbd6-eb94037438f7","Type":"ContainerStarted","Data":"a09fb08078aa6dc66491083f8e8149a2d37b238a73321f73dce14ca4d38842e4"}
Jul 17 00:52:53.597550 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:53.597408    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" event={"ID":"3055d1c0-7573-4c25-bbd6-eb94037438f7","Type":"ContainerStarted","Data":"24f1ebbd6d6b3f5405a22bce2c9ad45341972bf79c27df754e31001a980c2be5"}
Jul 17 00:52:53.613281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:53.613231    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" podStartSLOduration=1.613214323 podStartE2EDuration="1.613214323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:52:53.611024186 +0000 UTC m=+2168.187176760" watchObservedRunningTime="2026-07-17 00:52:53.613214323 +0000 UTC m=+2168.189366897"
Jul 17 00:52:54.323063 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.323024    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"]
Jul 17 00:52:54.467450 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.467423    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:52:54.594839 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.594757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls\") pod \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") "
Jul 17 00:52:54.594961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.594844    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dfkz\" (UniqueName: \"kubernetes.io/projected/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-kube-api-access-2dfkz\") pod \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") "
Jul 17 00:52:54.594961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.594884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-openshift-service-ca\") pod \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\" (UID: \"62a8b6e4-50ba-4691-933d-5d1c4f8d2003\") "
Jul 17 00:52:54.595260 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.595230    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62a8b6e4-50ba-4691-933d-5d1c4f8d2003" (UID: "62a8b6e4-50ba-4691-933d-5d1c4f8d2003"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:52:54.596635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.596610    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62a8b6e4-50ba-4691-933d-5d1c4f8d2003" (UID: "62a8b6e4-50ba-4691-933d-5d1c4f8d2003"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:52:54.596886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.596861    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-kube-api-access-2dfkz" (OuterVolumeSpecName: "kube-api-access-2dfkz") pod "62a8b6e4-50ba-4691-933d-5d1c4f8d2003" (UID: "62a8b6e4-50ba-4691-933d-5d1c4f8d2003"). InnerVolumeSpecName "kube-api-access-2dfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:52:54.604086 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.604062    2573 generic.go:356] "Generic (PLEG): container finished" podID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerID="a09fb08078aa6dc66491083f8e8149a2d37b238a73321f73dce14ca4d38842e4" exitCode=6
Jul 17 00:52:54.604185 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.604124    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" event={"ID":"3055d1c0-7573-4c25-bbd6-eb94037438f7","Type":"ContainerDied","Data":"a09fb08078aa6dc66491083f8e8149a2d37b238a73321f73dce14ca4d38842e4"}
Jul 17 00:52:54.604364 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.604345    2573 scope.go:117] "RemoveContainer" containerID="a09fb08078aa6dc66491083f8e8149a2d37b238a73321f73dce14ca4d38842e4"
Jul 17 00:52:54.605566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.605551    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"
Jul 17 00:52:54.605661 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.605598    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8" event={"ID":"62a8b6e4-50ba-4691-933d-5d1c4f8d2003","Type":"ContainerDied","Data":"dc3e6d8441f2841cad0da562842b92180b46d7f3de3076accc0a6efe3e697dcb"}
Jul 17 00:52:54.605661 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.605656    2573 scope.go:117] "RemoveContainer" containerID="b8ab4164a1abda6fea0e26bf841885e9e80ca62f41f594303571506a89207745"
Jul 17 00:52:54.629335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.629317    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"]
Jul 17 00:52:54.634654 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.634634    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4692a03f-8657966bf9-5xdj8"]
Jul 17 00:52:54.695430 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.695405    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:52:54.695430 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.695427    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:52:54.695614 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:54.695438    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfkz\" (UniqueName: \"kubernetes.io/projected/62a8b6e4-50ba-4691-933d-5d1c4f8d2003-kube-api-access-2dfkz\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:52:55.612362 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:55.612291    2573 generic.go:356] "Generic (PLEG): container finished" podID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerID="4dc00167fb9dab1a1ff0df9e6a4a8bab1111e463a2aa2e57919fc6dd73ebffec" exitCode=6
Jul 17 00:52:55.612770 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:55.612368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" event={"ID":"3055d1c0-7573-4c25-bbd6-eb94037438f7","Type":"ContainerDied","Data":"4dc00167fb9dab1a1ff0df9e6a4a8bab1111e463a2aa2e57919fc6dd73ebffec"}
Jul 17 00:52:55.612770 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:55.612424    2573 scope.go:117] "RemoveContainer" containerID="a09fb08078aa6dc66491083f8e8149a2d37b238a73321f73dce14ca4d38842e4"
Jul 17 00:52:55.612770 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:55.612632    2573 scope.go:117] "RemoveContainer" containerID="4dc00167fb9dab1a1ff0df9e6a4a8bab1111e463a2aa2e57919fc6dd73ebffec"
Jul 17 00:52:55.612948 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:55.612867    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5_odh-ai-gateway-infra(3055d1c0-7573-4c25-bbd6-eb94037438f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7"
Jul 17 00:52:55.973309 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:55.973218    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" path="/var/lib/kubelet/pods/62a8b6e4-50ba-4691-933d-5d1c4f8d2003/volumes"
Jul 17 00:52:56.621015 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:52:56.620991    2573 scope.go:117] "RemoveContainer" containerID="4dc00167fb9dab1a1ff0df9e6a4a8bab1111e463a2aa2e57919fc6dd73ebffec"
Jul 17 00:52:56.621386 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:52:56.621158    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5_odh-ai-gateway-infra(3055d1c0-7573-4c25-bbd6-eb94037438f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7"
Jul 17 00:53:01.502971 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.502936    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"]
Jul 17 00:53:01.503455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503436    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503457    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503472    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503477    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503492    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503498    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503512    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503518    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503524    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503529    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503537    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503542    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503550    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503554    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503607    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503615    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503621    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503627    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503634    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b626b3-acb6-4915-9f12-c8bac5c18395" containerName="revoke-keys"
Jul 17 00:53:01.503701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.503640    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="97da358a-2be9-4623-8c66-c477c7248d37" containerName="revoke-keys"
Jul 17 00:53:01.508132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.508113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.510086 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.510066    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-732542a6-serving-cert\""
Jul 17 00:53:01.510191 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.510126    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 00:53:01.527783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.527757    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"]
Jul 17 00:53:01.649448 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.649412    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.649624 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.649461    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsqtf\" (UniqueName: \"kubernetes.io/projected/7e3af469-52c3-4228-bc6f-ee97a64ace7e-kube-api-access-bsqtf\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.649624 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.649487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3af469-52c3-4228-bc6f-ee97a64ace7e-openshift-service-ca\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.750768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.750733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsqtf\" (UniqueName: \"kubernetes.io/projected/7e3af469-52c3-4228-bc6f-ee97a64ace7e-kube-api-access-bsqtf\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.750768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.750768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3af469-52c3-4228-bc6f-ee97a64ace7e-openshift-service-ca\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.750990 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.750874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.750990 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:01.750959    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-serving-cert: secret "maas-api-e2e-derive-732542a6-serving-cert" not found
Jul 17 00:53:01.751072 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:01.751010    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls podName:7e3af469-52c3-4228-bc6f-ee97a64ace7e nodeName:}" failed. No retries permitted until 2026-07-17 00:53:02.250994616 +0000 UTC m=+2176.827147168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls") pod "maas-api-e2e-derive-732542a6-bb64878bf-pktwf" (UID: "7e3af469-52c3-4228-bc6f-ee97a64ace7e") : secret "maas-api-e2e-derive-732542a6-serving-cert" not found
Jul 17 00:53:01.751468 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.751446    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3af469-52c3-4228-bc6f-ee97a64ace7e-openshift-service-ca\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:01.758691 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:01.758669    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsqtf\" (UniqueName: \"kubernetes.io/projected/7e3af469-52c3-4228-bc6f-ee97a64ace7e-kube-api-access-bsqtf\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:02.255554 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:02.255431    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:02.257988 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:02.257955    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls\") pod \"maas-api-e2e-derive-732542a6-bb64878bf-pktwf\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:02.419294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:02.419262    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:02.550740 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:02.550714    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"]
Jul 17 00:53:02.552763 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:53:02.552739    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3af469_52c3_4228_bc6f_ee97a64ace7e.slice/crio-83ea409f14560ddf32a310b088fcfd509651f5bfdcaaf593068e94f79b75df0f WatchSource:0}: Error finding container 83ea409f14560ddf32a310b088fcfd509651f5bfdcaaf593068e94f79b75df0f: Status 404 returned error can't find the container with id 83ea409f14560ddf32a310b088fcfd509651f5bfdcaaf593068e94f79b75df0f
Jul 17 00:53:02.554461 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:02.554443    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 00:53:02.653670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:02.653637    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerStarted","Data":"83ea409f14560ddf32a310b088fcfd509651f5bfdcaaf593068e94f79b75df0f"}
Jul 17 00:53:03.660324 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:03.660289    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerStarted","Data":"5510f2e61c535b0299e84e86b9a27087b2c770b52e3062ecff5008c30596cdb7"}
Jul 17 00:53:03.660708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:03.660469    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:03.676561 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:03.676520    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podStartSLOduration=1.788592341 podStartE2EDuration="2.676506582s" totalImagesPullingTime="887.914241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:53:03.674335563 +0000 UTC m=+2178.250488137" watchObservedRunningTime="2026-07-17 00:53:03.676506582 +0000 UTC m=+2178.252659156"
Jul 17 00:53:04.667147 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:04.667115    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerID="5510f2e61c535b0299e84e86b9a27087b2c770b52e3062ecff5008c30596cdb7" exitCode=1
Jul 17 00:53:04.667617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:04.667181    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerDied","Data":"5510f2e61c535b0299e84e86b9a27087b2c770b52e3062ecff5008c30596cdb7"}
Jul 17 00:53:04.667617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:04.667460    2573 scope.go:117] "RemoveContainer" containerID="5510f2e61c535b0299e84e86b9a27087b2c770b52e3062ecff5008c30596cdb7"
Jul 17 00:53:05.673844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.673755    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerID="df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574" exitCode=1
Jul 17 00:53:05.673844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.673823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerDied","Data":"df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574"}
Jul 17 00:53:05.674326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.673879    2573 scope.go:117] "RemoveContainer" containerID="5510f2e61c535b0299e84e86b9a27087b2c770b52e3062ecff5008c30596cdb7"
Jul 17 00:53:05.674326 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.674031    2573 scope.go:117] "RemoveContainer" containerID="df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574"
Jul 17 00:53:05.674326 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:05.674231    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:05.843853 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.843821    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"]
Jul 17 00:53:05.844242 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.844227    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:05.844296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.844244    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:05.844337 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.844301    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:05.844337 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.844313    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:05.848606 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.848577    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"
Jul 17 00:53:05.853369 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.853340    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"]
Jul 17 00:53:05.988221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:05.988154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bbd454-d3e4-40b5-ad33-c8886fbf3189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-z6rl6\" (UID: \"d7bbd454-d3e4-40b5-ad33-c8886fbf3189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"
Jul 17 00:53:06.088729 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.088702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bbd454-d3e4-40b5-ad33-c8886fbf3189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-z6rl6\" (UID: \"d7bbd454-d3e4-40b5-ad33-c8886fbf3189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"
Jul 17 00:53:06.089506 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.089482    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bbd454-d3e4-40b5-ad33-c8886fbf3189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-z6rl6\" (UID: \"d7bbd454-d3e4-40b5-ad33-c8886fbf3189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"
Jul 17 00:53:06.175415 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.175392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"
Jul 17 00:53:06.307376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.307342    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"]
Jul 17 00:53:06.308688 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:53:06.308662    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bbd454_d3e4_40b5_ad33_c8886fbf3189.slice/crio-cf4beb679cc5b04d193897d6c9efb090c8c3a3a738ea8e8b8ed7a4cdc16a74a5 WatchSource:0}: Error finding container cf4beb679cc5b04d193897d6c9efb090c8c3a3a738ea8e8b8ed7a4cdc16a74a5: Status 404 returned error can't find the container with id cf4beb679cc5b04d193897d6c9efb090c8c3a3a738ea8e8b8ed7a4cdc16a74a5
Jul 17 00:53:06.682630 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.682606    2573 scope.go:117] "RemoveContainer" containerID="df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574"
Jul 17 00:53:06.683074 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:06.682858    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:06.683994 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.683964    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" event={"ID":"d7bbd454-d3e4-40b5-ad33-c8886fbf3189","Type":"ContainerStarted","Data":"2de9dce9d85ed9e046bd750609c9a8be21298e9b73e74ff0e0b07650915ce6e2"}
Jul 17 00:53:06.684106 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.684023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" event={"ID":"d7bbd454-d3e4-40b5-ad33-c8886fbf3189","Type":"ContainerStarted","Data":"cf4beb679cc5b04d193897d6c9efb090c8c3a3a738ea8e8b8ed7a4cdc16a74a5"}
Jul 17 00:53:06.710185 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:06.710139    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" podStartSLOduration=1.7101266979999998 podStartE2EDuration="1.710126698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:53:06.708837349 +0000 UTC m=+2181.284989924" watchObservedRunningTime="2026-07-17 00:53:06.710126698 +0000 UTC m=+2181.286279271"
Jul 17 00:53:07.691575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:07.691541    2573 generic.go:356] "Generic (PLEG): container finished" podID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerID="2de9dce9d85ed9e046bd750609c9a8be21298e9b73e74ff0e0b07650915ce6e2" exitCode=7
Jul 17 00:53:07.692072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:07.691631    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" event={"ID":"d7bbd454-d3e4-40b5-ad33-c8886fbf3189","Type":"ContainerDied","Data":"2de9dce9d85ed9e046bd750609c9a8be21298e9b73e74ff0e0b07650915ce6e2"}
Jul 17 00:53:07.692072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:07.691852    2573 scope.go:117] "RemoveContainer" containerID="2de9dce9d85ed9e046bd750609c9a8be21298e9b73e74ff0e0b07650915ce6e2"
Jul 17 00:53:07.963053 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:07.963028    2573 scope.go:117] "RemoveContainer" containerID="4dc00167fb9dab1a1ff0df9e6a4a8bab1111e463a2aa2e57919fc6dd73ebffec"
Jul 17 00:53:08.660552 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.660531    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:08.660894 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.660880    2573 scope.go:117] "RemoveContainer" containerID="df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574"
Jul 17 00:53:08.661050 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:08.661031    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:08.699253 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.699227    2573 generic.go:356] "Generic (PLEG): container finished" podID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerID="fbaea43e3d160526b41c6f1b915057a96a786e36d951958a45ce8ea9e7e660a3" exitCode=7
Jul 17 00:53:08.699569 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.699302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" event={"ID":"d7bbd454-d3e4-40b5-ad33-c8886fbf3189","Type":"ContainerDied","Data":"fbaea43e3d160526b41c6f1b915057a96a786e36d951958a45ce8ea9e7e660a3"}
Jul 17 00:53:08.699569 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.699357    2573 scope.go:117] "RemoveContainer" containerID="2de9dce9d85ed9e046bd750609c9a8be21298e9b73e74ff0e0b07650915ce6e2"
Jul 17 00:53:08.699655 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.699579    2573 scope.go:117] "RemoveContainer" containerID="fbaea43e3d160526b41c6f1b915057a96a786e36d951958a45ce8ea9e7e660a3"
Jul 17 00:53:08.699779 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:08.699751    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-z6rl6_odh-ai-gateway-infra(d7bbd454-d3e4-40b5-ad33-c8886fbf3189)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189"
Jul 17 00:53:08.701150 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.701131    2573 generic.go:356] "Generic (PLEG): container finished" podID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerID="deeb174b1686bcf8460aafb3a196485166e09502f3d56b167de39055ff33a869" exitCode=6
Jul 17 00:53:08.701241 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.701166    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" event={"ID":"3055d1c0-7573-4c25-bbd6-eb94037438f7","Type":"ContainerDied","Data":"deeb174b1686bcf8460aafb3a196485166e09502f3d56b167de39055ff33a869"}
Jul 17 00:53:08.701399 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.701385    2573 scope.go:117] "RemoveContainer" containerID="deeb174b1686bcf8460aafb3a196485166e09502f3d56b167de39055ff33a869"
Jul 17 00:53:08.701581 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:08.701563    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5_odh-ai-gateway-infra(3055d1c0-7573-4c25-bbd6-eb94037438f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7"
Jul 17 00:53:08.716082 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:08.716058    2573 scope.go:117] "RemoveContainer" containerID="4dc00167fb9dab1a1ff0df9e6a4a8bab1111e463a2aa2e57919fc6dd73ebffec"
Jul 17 00:53:09.707783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:09.707753    2573 scope.go:117] "RemoveContainer" containerID="fbaea43e3d160526b41c6f1b915057a96a786e36d951958a45ce8ea9e7e660a3"
Jul 17 00:53:09.708257 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:09.707986    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-z6rl6_odh-ai-gateway-infra(d7bbd454-d3e4-40b5-ad33-c8886fbf3189)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189"
Jul 17 00:53:09.738387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:09.738364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"]
Jul 17 00:53:09.871004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:09.870983    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"
Jul 17 00:53:10.019517 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.019493    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3055d1c0-7573-4c25-bbd6-eb94037438f7-maas-api-service-ca\") pod \"3055d1c0-7573-4c25-bbd6-eb94037438f7\" (UID: \"3055d1c0-7573-4c25-bbd6-eb94037438f7\") "
Jul 17 00:53:10.019887 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.019867    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3055d1c0-7573-4c25-bbd6-eb94037438f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3055d1c0-7573-4c25-bbd6-eb94037438f7" (UID: "3055d1c0-7573-4c25-bbd6-eb94037438f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:53:10.120243 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.120207    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3055d1c0-7573-4c25-bbd6-eb94037438f7-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:53:10.391076 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391006    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"]
Jul 17 00:53:10.391388 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391375    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:10.391446 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391389    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:10.391446 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391408    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:10.391446 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391414    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:10.391555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391465    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:10.391555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391474    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:10.391555 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.391483    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a8b6e4-50ba-4691-933d-5d1c4f8d2003" containerName="maas-api"
Jul 17 00:53:10.394467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.394446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"
Jul 17 00:53:10.400589 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.400563    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"]
Jul 17 00:53:10.523436 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.523406    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1869bb4c-c26f-4604-8e26-3887f99dccd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg\" (UID: \"1869bb4c-c26f-4604-8e26-3887f99dccd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"
Jul 17 00:53:10.624725 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.624702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1869bb4c-c26f-4604-8e26-3887f99dccd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg\" (UID: \"1869bb4c-c26f-4604-8e26-3887f99dccd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"
Jul 17 00:53:10.625249 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.625233    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1869bb4c-c26f-4604-8e26-3887f99dccd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg\" (UID: \"1869bb4c-c26f-4604-8e26-3887f99dccd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"
Jul 17 00:53:10.707106 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.707052    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"
Jul 17 00:53:10.715246 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.715222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5" event={"ID":"3055d1c0-7573-4c25-bbd6-eb94037438f7","Type":"ContainerDied","Data":"24f1ebbd6d6b3f5405a22bce2c9ad45341972bf79c27df754e31001a980c2be5"}
Jul 17 00:53:10.715611 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.715282    2573 scope.go:117] "RemoveContainer" containerID="deeb174b1686bcf8460aafb3a196485166e09502f3d56b167de39055ff33a869"
Jul 17 00:53:10.715611 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.715231    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"
Jul 17 00:53:10.744279 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.744252    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"]
Jul 17 00:53:10.750204 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.750176    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-xf2p5"]
Jul 17 00:53:10.839876 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:10.839852    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"]
Jul 17 00:53:10.841460 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:53:10.841427    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1869bb4c_c26f_4604_8e26_3887f99dccd0.slice/crio-e6b783f6b050cab4b0ee6c0ea724b7082b018dfc3187e49c013ea605187fdc45 WatchSource:0}: Error finding container e6b783f6b050cab4b0ee6c0ea724b7082b018dfc3187e49c013ea605187fdc45: Status 404 returned error can't find the container with id e6b783f6b050cab4b0ee6c0ea724b7082b018dfc3187e49c013ea605187fdc45
Jul 17 00:53:11.722378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:11.722345    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerID="7317d83cef27206cab96981c2488ccde877514cd473e23970aca2671954339e9" exitCode=6
Jul 17 00:53:11.722783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:11.722440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" event={"ID":"1869bb4c-c26f-4604-8e26-3887f99dccd0","Type":"ContainerDied","Data":"7317d83cef27206cab96981c2488ccde877514cd473e23970aca2671954339e9"}
Jul 17 00:53:11.722783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:11.722505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" event={"ID":"1869bb4c-c26f-4604-8e26-3887f99dccd0","Type":"ContainerStarted","Data":"e6b783f6b050cab4b0ee6c0ea724b7082b018dfc3187e49c013ea605187fdc45"}
Jul 17 00:53:11.722783 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:11.722579    2573 scope.go:117] "RemoveContainer" containerID="7317d83cef27206cab96981c2488ccde877514cd473e23970aca2671954339e9"
Jul 17 00:53:11.969605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:11.969581    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" path="/var/lib/kubelet/pods/3055d1c0-7573-4c25-bbd6-eb94037438f7/volumes"
Jul 17 00:53:12.420245 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:12.420219    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:12.420620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:12.420607    2573 scope.go:117] "RemoveContainer" containerID="df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574"
Jul 17 00:53:12.420779 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:12.420759    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:12.730831 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:12.730780    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerID="b22a6e25f84af5b29e9d3b6975b8644e22f076bdc4dc627cdd2a5474ebca2f8e" exitCode=6
Jul 17 00:53:12.731310 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:12.730822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" event={"ID":"1869bb4c-c26f-4604-8e26-3887f99dccd0","Type":"ContainerDied","Data":"b22a6e25f84af5b29e9d3b6975b8644e22f076bdc4dc627cdd2a5474ebca2f8e"}
Jul 17 00:53:12.731310 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:12.730887    2573 scope.go:117] "RemoveContainer" containerID="7317d83cef27206cab96981c2488ccde877514cd473e23970aca2671954339e9"
Jul 17 00:53:12.731310 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:12.731115    2573 scope.go:117] "RemoveContainer" containerID="b22a6e25f84af5b29e9d3b6975b8644e22f076bdc4dc627cdd2a5474ebca2f8e"
Jul 17 00:53:12.731457 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:12.731360    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg_odh-ai-gateway-infra(1869bb4c-c26f-4604-8e26-3887f99dccd0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0"
Jul 17 00:53:13.737911 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:13.737879    2573 scope.go:117] "RemoveContainer" containerID="b22a6e25f84af5b29e9d3b6975b8644e22f076bdc4dc627cdd2a5474ebca2f8e"
Jul 17 00:53:13.738300 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:13.738093    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg_odh-ai-gateway-infra(1869bb4c-c26f-4604-8e26-3887f99dccd0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0"
Jul 17 00:53:18.569544 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.569511    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"]
Jul 17 00:53:18.569949 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.569887    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:18.569949 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.569899    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:18.570041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.569976    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3055d1c0-7573-4c25-bbd6-eb94037438f7" containerName="revoke-keys"
Jul 17 00:53:18.579524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.579490    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"
Jul 17 00:53:18.580778 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.580747    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"]
Jul 17 00:53:18.691954 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.691925    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76f1fe8-e277-4fdd-b7eb-3864a60d376e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6\" (UID: \"a76f1fe8-e277-4fdd-b7eb-3864a60d376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"
Jul 17 00:53:18.792295 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.792258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76f1fe8-e277-4fdd-b7eb-3864a60d376e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6\" (UID: \"a76f1fe8-e277-4fdd-b7eb-3864a60d376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"
Jul 17 00:53:18.792884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.792866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76f1fe8-e277-4fdd-b7eb-3864a60d376e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6\" (UID: \"a76f1fe8-e277-4fdd-b7eb-3864a60d376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"
Jul 17 00:53:18.890972 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:18.890916    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"
Jul 17 00:53:19.023569 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:19.023539    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"]
Jul 17 00:53:19.024873 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:53:19.024847    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76f1fe8_e277_4fdd_b7eb_3864a60d376e.slice/crio-91673829966bb4e7da757f1528169821edd283021d26bbbf07eb067b3eed6836 WatchSource:0}: Error finding container 91673829966bb4e7da757f1528169821edd283021d26bbbf07eb067b3eed6836: Status 404 returned error can't find the container with id 91673829966bb4e7da757f1528169821edd283021d26bbbf07eb067b3eed6836
Jul 17 00:53:19.773481 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:19.773453    2573 generic.go:356] "Generic (PLEG): container finished" podID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerID="df799211b788439fcf94a6b39eec99f5812d25dbea404fc6763459e5ee861f3e" exitCode=6
Jul 17 00:53:19.773806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:19.773538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" event={"ID":"a76f1fe8-e277-4fdd-b7eb-3864a60d376e","Type":"ContainerDied","Data":"df799211b788439fcf94a6b39eec99f5812d25dbea404fc6763459e5ee861f3e"}
Jul 17 00:53:19.773806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:19.773596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" event={"ID":"a76f1fe8-e277-4fdd-b7eb-3864a60d376e","Type":"ContainerStarted","Data":"91673829966bb4e7da757f1528169821edd283021d26bbbf07eb067b3eed6836"}
Jul 17 00:53:19.773806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:19.773655    2573 scope.go:117] "RemoveContainer" containerID="df799211b788439fcf94a6b39eec99f5812d25dbea404fc6763459e5ee861f3e"
Jul 17 00:53:20.780263 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:20.780235    2573 generic.go:356] "Generic (PLEG): container finished" podID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerID="e88dae8a783ac027030df15435ac37b2b6ed9f1715fb3e65da917aeb25e2ba96" exitCode=6
Jul 17 00:53:20.780692 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:20.780276    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" event={"ID":"a76f1fe8-e277-4fdd-b7eb-3864a60d376e","Type":"ContainerDied","Data":"e88dae8a783ac027030df15435ac37b2b6ed9f1715fb3e65da917aeb25e2ba96"}
Jul 17 00:53:20.780692 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:20.780324    2573 scope.go:117] "RemoveContainer" containerID="df799211b788439fcf94a6b39eec99f5812d25dbea404fc6763459e5ee861f3e"
Jul 17 00:53:20.780692 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:20.780576    2573 scope.go:117] "RemoveContainer" containerID="e88dae8a783ac027030df15435ac37b2b6ed9f1715fb3e65da917aeb25e2ba96"
Jul 17 00:53:20.780839 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:20.780824    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6_odh-ai-gateway-infra(a76f1fe8-e277-4fdd-b7eb-3864a60d376e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e"
Jul 17 00:53:21.786947 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:21.786919    2573 scope.go:117] "RemoveContainer" containerID="e88dae8a783ac027030df15435ac37b2b6ed9f1715fb3e65da917aeb25e2ba96"
Jul 17 00:53:21.787349 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:21.787093    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6_odh-ai-gateway-infra(a76f1fe8-e277-4fdd-b7eb-3864a60d376e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e"
Jul 17 00:53:21.962574 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:21.962543    2573 scope.go:117] "RemoveContainer" containerID="fbaea43e3d160526b41c6f1b915057a96a786e36d951958a45ce8ea9e7e660a3"
Jul 17 00:53:22.794236 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:22.794171    2573 generic.go:356] "Generic (PLEG): container finished" podID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerID="ee6b845adbaf18926197d65d57b3a7694adf00f1f314d1fb8ff834bf90995549" exitCode=7
Jul 17 00:53:22.794236 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:22.794220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" event={"ID":"d7bbd454-d3e4-40b5-ad33-c8886fbf3189","Type":"ContainerDied","Data":"ee6b845adbaf18926197d65d57b3a7694adf00f1f314d1fb8ff834bf90995549"}
Jul 17 00:53:22.794632 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:22.794283    2573 scope.go:117] "RemoveContainer" containerID="fbaea43e3d160526b41c6f1b915057a96a786e36d951958a45ce8ea9e7e660a3"
Jul 17 00:53:22.794632 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:22.794523    2573 scope.go:117] "RemoveContainer" containerID="ee6b845adbaf18926197d65d57b3a7694adf00f1f314d1fb8ff834bf90995549"
Jul 17 00:53:22.794722 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:22.794703    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-z6rl6_odh-ai-gateway-infra(d7bbd454-d3e4-40b5-ad33-c8886fbf3189)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189"
Jul 17 00:53:23.818195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:23.818160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"]
Jul 17 00:53:23.955702 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:23.955677    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"
Jul 17 00:53:24.130437 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.130352    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bbd454-d3e4-40b5-ad33-c8886fbf3189-maas-api-service-ca\") pod \"d7bbd454-d3e4-40b5-ad33-c8886fbf3189\" (UID: \"d7bbd454-d3e4-40b5-ad33-c8886fbf3189\") "
Jul 17 00:53:24.130771 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.130750    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7bbd454-d3e4-40b5-ad33-c8886fbf3189-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7bbd454-d3e4-40b5-ad33-c8886fbf3189" (UID: "d7bbd454-d3e4-40b5-ad33-c8886fbf3189"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:53:24.232423 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.232388    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bbd454-d3e4-40b5-ad33-c8886fbf3189-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:53:24.807490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.807454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6" event={"ID":"d7bbd454-d3e4-40b5-ad33-c8886fbf3189","Type":"ContainerDied","Data":"cf4beb679cc5b04d193897d6c9efb090c8c3a3a738ea8e8b8ed7a4cdc16a74a5"}
Jul 17 00:53:24.807490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.807484    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"
Jul 17 00:53:24.807712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.807547    2573 scope.go:117] "RemoveContainer" containerID="ee6b845adbaf18926197d65d57b3a7694adf00f1f314d1fb8ff834bf90995549"
Jul 17 00:53:24.829329 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.829304    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"]
Jul 17 00:53:24.834032 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.834011    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-z6rl6"]
Jul 17 00:53:24.962362 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:24.962337    2573 scope.go:117] "RemoveContainer" containerID="df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574"
Jul 17 00:53:25.815021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:25.814996    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerID="0a11fa9cc12037a9580b08507ec8771dd0e0511402ec31a71536d027bf18dda8" exitCode=1
Jul 17 00:53:25.815131 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:25.815066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerDied","Data":"0a11fa9cc12037a9580b08507ec8771dd0e0511402ec31a71536d027bf18dda8"}
Jul 17 00:53:25.815131 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:25.815121    2573 scope.go:117] "RemoveContainer" containerID="df966668ea0f499662a0711f7aced0d252a7fb1ceedfd97330793abfbd16e574"
Jul 17 00:53:25.815438 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:25.815422    2573 scope.go:117] "RemoveContainer" containerID="0a11fa9cc12037a9580b08507ec8771dd0e0511402ec31a71536d027bf18dda8"
Jul 17 00:53:25.815644 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:25.815625    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:25.968226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:25.968195    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" path="/var/lib/kubelet/pods/d7bbd454-d3e4-40b5-ad33-c8886fbf3189/volumes"
Jul 17 00:53:28.661287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:28.661242    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:28.661818 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:28.661682    2573 scope.go:117] "RemoveContainer" containerID="0a11fa9cc12037a9580b08507ec8771dd0e0511402ec31a71536d027bf18dda8"
Jul 17 00:53:28.661942 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:28.661920    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:28.962427 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:28.962347    2573 scope.go:117] "RemoveContainer" containerID="b22a6e25f84af5b29e9d3b6975b8644e22f076bdc4dc627cdd2a5474ebca2f8e"
Jul 17 00:53:29.840436 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:29.840354    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerID="319c2e2b3e959a503b721ef1b8aefd40df3e44be8c9fec0174f7d2283dae30df" exitCode=6
Jul 17 00:53:29.840436 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:29.840420    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" event={"ID":"1869bb4c-c26f-4604-8e26-3887f99dccd0","Type":"ContainerDied","Data":"319c2e2b3e959a503b721ef1b8aefd40df3e44be8c9fec0174f7d2283dae30df"}
Jul 17 00:53:29.840913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:29.840477    2573 scope.go:117] "RemoveContainer" containerID="b22a6e25f84af5b29e9d3b6975b8644e22f076bdc4dc627cdd2a5474ebca2f8e"
Jul 17 00:53:29.840913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:29.840754    2573 scope.go:117] "RemoveContainer" containerID="319c2e2b3e959a503b721ef1b8aefd40df3e44be8c9fec0174f7d2283dae30df"
Jul 17 00:53:29.841013 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:29.840991    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg_odh-ai-gateway-infra(1869bb4c-c26f-4604-8e26-3887f99dccd0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0"
Jul 17 00:53:30.864585 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:30.864547    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"]
Jul 17 00:53:31.000759 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.000737    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"
Jul 17 00:53:31.082931 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.082906    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1869bb4c-c26f-4604-8e26-3887f99dccd0-maas-api-service-ca\") pod \"1869bb4c-c26f-4604-8e26-3887f99dccd0\" (UID: \"1869bb4c-c26f-4604-8e26-3887f99dccd0\") "
Jul 17 00:53:31.083239 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.083213    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1869bb4c-c26f-4604-8e26-3887f99dccd0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1869bb4c-c26f-4604-8e26-3887f99dccd0" (UID: "1869bb4c-c26f-4604-8e26-3887f99dccd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:53:31.183865 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.183787    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1869bb4c-c26f-4604-8e26-3887f99dccd0-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:53:31.854443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.854411    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"
Jul 17 00:53:31.854594 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.854459    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg" event={"ID":"1869bb4c-c26f-4604-8e26-3887f99dccd0","Type":"ContainerDied","Data":"e6b783f6b050cab4b0ee6c0ea724b7082b018dfc3187e49c013ea605187fdc45"}
Jul 17 00:53:31.854594 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.854520    2573 scope.go:117] "RemoveContainer" containerID="319c2e2b3e959a503b721ef1b8aefd40df3e44be8c9fec0174f7d2283dae30df"
Jul 17 00:53:31.875949 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.875922    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"]
Jul 17 00:53:31.882086 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.882063    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-nsdmg"]
Jul 17 00:53:31.967912 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:31.967889    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" path="/var/lib/kubelet/pods/1869bb4c-c26f-4604-8e26-3887f99dccd0/volumes"
Jul 17 00:53:32.420340 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:32.420315    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:32.420655 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:32.420641    2573 scope.go:117] "RemoveContainer" containerID="0a11fa9cc12037a9580b08507ec8771dd0e0511402ec31a71536d027bf18dda8"
Jul 17 00:53:32.420834 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:32.420814    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:33.963167 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:33.963140    2573 scope.go:117] "RemoveContainer" containerID="e88dae8a783ac027030df15435ac37b2b6ed9f1715fb3e65da917aeb25e2ba96"
Jul 17 00:53:34.872799 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:34.872770    2573 generic.go:356] "Generic (PLEG): container finished" podID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerID="4e7558a282fd0cc8da2fdbe7c2910eff2bbeb8b3f3974eca61e97db7bd3131cb" exitCode=6
Jul 17 00:53:34.872937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:34.872842    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" event={"ID":"a76f1fe8-e277-4fdd-b7eb-3864a60d376e","Type":"ContainerDied","Data":"4e7558a282fd0cc8da2fdbe7c2910eff2bbeb8b3f3974eca61e97db7bd3131cb"}
Jul 17 00:53:34.872937 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:34.872902    2573 scope.go:117] "RemoveContainer" containerID="e88dae8a783ac027030df15435ac37b2b6ed9f1715fb3e65da917aeb25e2ba96"
Jul 17 00:53:34.873181 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:34.873160    2573 scope.go:117] "RemoveContainer" containerID="4e7558a282fd0cc8da2fdbe7c2910eff2bbeb8b3f3974eca61e97db7bd3131cb"
Jul 17 00:53:34.873361 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:34.873335    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6_odh-ai-gateway-infra(a76f1fe8-e277-4fdd-b7eb-3864a60d376e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e"
Jul 17 00:53:35.898259 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:35.898215    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"]
Jul 17 00:53:36.038862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.038834    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"
Jul 17 00:53:36.126139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.126112    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76f1fe8-e277-4fdd-b7eb-3864a60d376e-maas-api-service-ca\") pod \"a76f1fe8-e277-4fdd-b7eb-3864a60d376e\" (UID: \"a76f1fe8-e277-4fdd-b7eb-3864a60d376e\") "
Jul 17 00:53:36.126438 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.126417    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76f1fe8-e277-4fdd-b7eb-3864a60d376e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a76f1fe8-e277-4fdd-b7eb-3864a60d376e" (UID: "a76f1fe8-e277-4fdd-b7eb-3864a60d376e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:53:36.226780 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.226706    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76f1fe8-e277-4fdd-b7eb-3864a60d376e-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:53:36.887229 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.887201    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"
Jul 17 00:53:36.887410 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.887244    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6" event={"ID":"a76f1fe8-e277-4fdd-b7eb-3864a60d376e","Type":"ContainerDied","Data":"91673829966bb4e7da757f1528169821edd283021d26bbbf07eb067b3eed6836"}
Jul 17 00:53:36.887410 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.887302    2573 scope.go:117] "RemoveContainer" containerID="4e7558a282fd0cc8da2fdbe7c2910eff2bbeb8b3f3974eca61e97db7bd3131cb"
Jul 17 00:53:36.909743 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.909717    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"]
Jul 17 00:53:36.916856 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:36.916833    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pfhb6"]
Jul 17 00:53:37.968353 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:37.968319    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" path="/var/lib/kubelet/pods/a76f1fe8-e277-4fdd-b7eb-3864a60d376e/volumes"
Jul 17 00:53:42.886300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886265    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"]
Jul 17 00:53:42.886678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886632    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:42.886678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886643    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:42.886678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886655    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:42.886678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886661    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:42.886678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886667    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:42.886678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886672    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:42.886678 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886681    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886687    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886696    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886701    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886707    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886712    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886811    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886820    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886826    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886834    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886839    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:42.886936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.886845    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:42.890973 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.890955    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"
Jul 17 00:53:42.893435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.893415    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:53:42.904601 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.904571    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"]
Jul 17 00:53:42.986408 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:42.986377    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx\" (UID: \"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"
Jul 17 00:53:43.087860 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:43.087835    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx\" (UID: \"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"
Jul 17 00:53:43.088339 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:43.088323    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx\" (UID: \"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"
Jul 17 00:53:43.201640 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:43.201573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"
Jul 17 00:53:43.329484 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:43.329458    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"]
Jul 17 00:53:43.331672 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:53:43.331649    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a839a0f_6a3d_4abd_afa0_c4fb7e647b7a.slice/crio-1034ac7f849f2c901762e1c3b594277651e1c135e2fb2f322073f0c4e918cdb4 WatchSource:0}: Error finding container 1034ac7f849f2c901762e1c3b594277651e1c135e2fb2f322073f0c4e918cdb4: Status 404 returned error can't find the container with id 1034ac7f849f2c901762e1c3b594277651e1c135e2fb2f322073f0c4e918cdb4
Jul 17 00:53:43.928822 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:43.928725    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" event={"ID":"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a","Type":"ContainerStarted","Data":"b58f610f10caa7d276c9432b9e3cb7ee00ff5b6d611b5613d7ed53aba47c17b1"}
Jul 17 00:53:43.928822 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:43.928779    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" event={"ID":"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a","Type":"ContainerStarted","Data":"1034ac7f849f2c901762e1c3b594277651e1c135e2fb2f322073f0c4e918cdb4"}
Jul 17 00:53:43.943956 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:43.943918    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" podStartSLOduration=1.9439010319999999 podStartE2EDuration="1.943901032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:53:43.941570988 +0000 UTC m=+2218.517723564" watchObservedRunningTime="2026-07-17 00:53:43.943901032 +0000 UTC m=+2218.520053606"
Jul 17 00:53:44.935712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:44.935679    2573 generic.go:356] "Generic (PLEG): container finished" podID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerID="b58f610f10caa7d276c9432b9e3cb7ee00ff5b6d611b5613d7ed53aba47c17b1" exitCode=6
Jul 17 00:53:44.936101 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:44.935741    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" event={"ID":"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a","Type":"ContainerDied","Data":"b58f610f10caa7d276c9432b9e3cb7ee00ff5b6d611b5613d7ed53aba47c17b1"}
Jul 17 00:53:44.936101 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:44.935917    2573 scope.go:117] "RemoveContainer" containerID="b58f610f10caa7d276c9432b9e3cb7ee00ff5b6d611b5613d7ed53aba47c17b1"
Jul 17 00:53:45.943262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:45.943178    2573 generic.go:356] "Generic (PLEG): container finished" podID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerID="43d22e418a12005b44c620912de81ae4fd8f33ee5c99a9c78eea8378e7799a1e" exitCode=6
Jul 17 00:53:45.943262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:45.943230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" event={"ID":"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a","Type":"ContainerDied","Data":"43d22e418a12005b44c620912de81ae4fd8f33ee5c99a9c78eea8378e7799a1e"}
Jul 17 00:53:45.943744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:45.943277    2573 scope.go:117] "RemoveContainer" containerID="b58f610f10caa7d276c9432b9e3cb7ee00ff5b6d611b5613d7ed53aba47c17b1"
Jul 17 00:53:45.943744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:45.943460    2573 scope.go:117] "RemoveContainer" containerID="43d22e418a12005b44c620912de81ae4fd8f33ee5c99a9c78eea8378e7799a1e"
Jul 17 00:53:45.943744 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:45.943679    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx_odh-ai-gateway-infra(8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a"
Jul 17 00:53:46.950376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:46.950349    2573 scope.go:117] "RemoveContainer" containerID="43d22e418a12005b44c620912de81ae4fd8f33ee5c99a9c78eea8378e7799a1e"
Jul 17 00:53:46.950750 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:46.950530    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx_odh-ai-gateway-infra(8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a"
Jul 17 00:53:47.962780 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:47.962745    2573 scope.go:117] "RemoveContainer" containerID="0a11fa9cc12037a9580b08507ec8771dd0e0511402ec31a71536d027bf18dda8"
Jul 17 00:53:48.962985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:48.962912    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd" exitCode=1
Jul 17 00:53:48.963373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:48.962992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerDied","Data":"0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"}
Jul 17 00:53:48.963373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:48.963068    2573 scope.go:117] "RemoveContainer" containerID="0a11fa9cc12037a9580b08507ec8771dd0e0511402ec31a71536d027bf18dda8"
Jul 17 00:53:48.963488 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:48.963408    2573 scope.go:117] "RemoveContainer" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"
Jul 17 00:53:48.963610 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:48.963587    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:52.420075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:52.420042    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:52.420452 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:52.420361    2573 scope.go:117] "RemoveContainer" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"
Jul 17 00:53:52.420563 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:52.420543    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:53.660623 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:53.660594    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:53:53.661026 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:53.661015    2573 scope.go:117] "RemoveContainer" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"
Jul 17 00:53:53.661202 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:53.661181    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:53:55.887327 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.887293    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"]
Jul 17 00:53:55.887880 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.887860    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:55.887982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.887884    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:55.887982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.887907    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:55.887982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.887916    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:55.887982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.887926    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:55.887982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.887938    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:55.888143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.888032    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7bbd454-d3e4-40b5-ad33-c8886fbf3189" containerName="revoke-keys"
Jul 17 00:53:55.888143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.888042    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76f1fe8-e277-4fdd-b7eb-3864a60d376e" containerName="revoke-keys"
Jul 17 00:53:55.888143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.888052    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1869bb4c-c26f-4604-8e26-3887f99dccd0" containerName="revoke-keys"
Jul 17 00:53:55.891588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.891568    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"
Jul 17 00:53:55.896177 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.896156    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"]
Jul 17 00:53:55.990045 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:55.990014    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276ab6b5-f9c9-4567-8e86-47ecec158461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5p7vf\" (UID: \"276ab6b5-f9c9-4567-8e86-47ecec158461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"
Jul 17 00:53:56.091443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:56.091414    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276ab6b5-f9c9-4567-8e86-47ecec158461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5p7vf\" (UID: \"276ab6b5-f9c9-4567-8e86-47ecec158461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"
Jul 17 00:53:56.092021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:56.092002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276ab6b5-f9c9-4567-8e86-47ecec158461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5p7vf\" (UID: \"276ab6b5-f9c9-4567-8e86-47ecec158461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"
Jul 17 00:53:56.204900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:56.204823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"
Jul 17 00:53:56.338739 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:56.338714    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"]
Jul 17 00:53:56.339066 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:53:56.339042    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276ab6b5_f9c9_4567_8e86_47ecec158461.slice/crio-f4b0e48c34e73bbb8306dd57d7bcd9ae9985794c04de1d741c319d620a4d5625 WatchSource:0}: Error finding container f4b0e48c34e73bbb8306dd57d7bcd9ae9985794c04de1d741c319d620a4d5625: Status 404 returned error can't find the container with id f4b0e48c34e73bbb8306dd57d7bcd9ae9985794c04de1d741c319d620a4d5625
Jul 17 00:53:57.008630 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:57.008599    2573 generic.go:356] "Generic (PLEG): container finished" podID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerID="9898cd744414944ee294b873eaa809c1ecd5f5a9781bac1e8bb5dd40fb445fd3" exitCode=7
Jul 17 00:53:57.009028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:57.008635    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" event={"ID":"276ab6b5-f9c9-4567-8e86-47ecec158461","Type":"ContainerDied","Data":"9898cd744414944ee294b873eaa809c1ecd5f5a9781bac1e8bb5dd40fb445fd3"}
Jul 17 00:53:57.009028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:57.008697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" event={"ID":"276ab6b5-f9c9-4567-8e86-47ecec158461","Type":"ContainerStarted","Data":"f4b0e48c34e73bbb8306dd57d7bcd9ae9985794c04de1d741c319d620a4d5625"}
Jul 17 00:53:57.009028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:57.008875    2573 scope.go:117] "RemoveContainer" containerID="9898cd744414944ee294b873eaa809c1ecd5f5a9781bac1e8bb5dd40fb445fd3"
Jul 17 00:53:57.962924 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:57.962860    2573 scope.go:117] "RemoveContainer" containerID="43d22e418a12005b44c620912de81ae4fd8f33ee5c99a9c78eea8378e7799a1e"
Jul 17 00:53:58.016047 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:58.016016    2573 generic.go:356] "Generic (PLEG): container finished" podID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerID="e85901f35f2ddcca63b1a152b93ed24c390b7d1b5fc4ff01382a09e662dcca6a" exitCode=7
Jul 17 00:53:58.016411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:58.016097    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" event={"ID":"276ab6b5-f9c9-4567-8e86-47ecec158461","Type":"ContainerDied","Data":"e85901f35f2ddcca63b1a152b93ed24c390b7d1b5fc4ff01382a09e662dcca6a"}
Jul 17 00:53:58.016411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:58.016158    2573 scope.go:117] "RemoveContainer" containerID="9898cd744414944ee294b873eaa809c1ecd5f5a9781bac1e8bb5dd40fb445fd3"
Jul 17 00:53:58.016411 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:58.016345    2573 scope.go:117] "RemoveContainer" containerID="e85901f35f2ddcca63b1a152b93ed24c390b7d1b5fc4ff01382a09e662dcca6a"
Jul 17 00:53:58.016550 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:58.016536    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5p7vf_odh-ai-gateway-infra(276ab6b5-f9c9-4567-8e86-47ecec158461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461"
Jul 17 00:53:59.023963 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:59.023931    2573 scope.go:117] "RemoveContainer" containerID="e85901f35f2ddcca63b1a152b93ed24c390b7d1b5fc4ff01382a09e662dcca6a"
Jul 17 00:53:59.024393 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:59.024137    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5p7vf_odh-ai-gateway-infra(276ab6b5-f9c9-4567-8e86-47ecec158461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461"
Jul 17 00:53:59.025453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:59.025428    2573 generic.go:356] "Generic (PLEG): container finished" podID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerID="1207c8da38540ba6d92f0b58a7e828cfcb3613d21390d2af485ad334768cd0ca" exitCode=6
Jul 17 00:53:59.025552 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:59.025458    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" event={"ID":"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a","Type":"ContainerDied","Data":"1207c8da38540ba6d92f0b58a7e828cfcb3613d21390d2af485ad334768cd0ca"}
Jul 17 00:53:59.025552 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:59.025530    2573 scope.go:117] "RemoveContainer" containerID="43d22e418a12005b44c620912de81ae4fd8f33ee5c99a9c78eea8378e7799a1e"
Jul 17 00:53:59.025804 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:53:59.025775    2573 scope.go:117] "RemoveContainer" containerID="1207c8da38540ba6d92f0b58a7e828cfcb3613d21390d2af485ad334768cd0ca"
Jul 17 00:53:59.025974 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:53:59.025953    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx_odh-ai-gateway-infra(8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a"
Jul 17 00:54:00.060221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:00.060186    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"]
Jul 17 00:54:00.209869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:00.209846    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"
Jul 17 00:54:00.324706 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:00.324611    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a-maas-api-service-ca\") pod \"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a\" (UID: \"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a\") "
Jul 17 00:54:00.325024 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:00.325003    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" (UID: "8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:54:00.426153 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:00.426126    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:54:01.038214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:01.038186    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"
Jul 17 00:54:01.038392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:01.038223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx" event={"ID":"8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a","Type":"ContainerDied","Data":"1034ac7f849f2c901762e1c3b594277651e1c135e2fb2f322073f0c4e918cdb4"}
Jul 17 00:54:01.038392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:01.038287    2573 scope.go:117] "RemoveContainer" containerID="1207c8da38540ba6d92f0b58a7e828cfcb3613d21390d2af485ad334768cd0ca"
Jul 17 00:54:01.058606 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:01.058582    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"]
Jul 17 00:54:01.061617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:01.061595    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-694sx"]
Jul 17 00:54:01.968588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:01.968556    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" path="/var/lib/kubelet/pods/8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a/volumes"
Jul 17 00:54:05.964956 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:05.964922    2573 scope.go:117] "RemoveContainer" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"
Jul 17 00:54:05.965344 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:05.965122    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:54:08.609466 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609383    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"]
Jul 17 00:54:08.609840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609738    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:08.609840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609749    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:08.609840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609762    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:08.609840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609767    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:08.609989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609850    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:08.609989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609862    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:08.609989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.609869    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:08.613965 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.613949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"
Jul 17 00:54:08.619037 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.619008    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"]
Jul 17 00:54:08.696024 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.696002    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f0c46-b1fc-4a23-b93d-f8142042e36d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c\" (UID: \"dc6f0c46-b1fc-4a23-b93d-f8142042e36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"
Jul 17 00:54:08.797373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.797345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f0c46-b1fc-4a23-b93d-f8142042e36d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c\" (UID: \"dc6f0c46-b1fc-4a23-b93d-f8142042e36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"
Jul 17 00:54:08.797982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.797963    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f0c46-b1fc-4a23-b93d-f8142042e36d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c\" (UID: \"dc6f0c46-b1fc-4a23-b93d-f8142042e36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"
Jul 17 00:54:08.926443 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:08.926390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"
Jul 17 00:54:09.053197 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:09.053166    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"]
Jul 17 00:54:09.054357 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:54:09.054331    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6f0c46_b1fc_4a23_b93d_f8142042e36d.slice/crio-28e217ea7bfba558b204bd78ff64ed4e67c6b0a649a2d7a61260cab4e4d293bd WatchSource:0}: Error finding container 28e217ea7bfba558b204bd78ff64ed4e67c6b0a649a2d7a61260cab4e4d293bd: Status 404 returned error can't find the container with id 28e217ea7bfba558b204bd78ff64ed4e67c6b0a649a2d7a61260cab4e4d293bd
Jul 17 00:54:09.084533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:09.084507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" event={"ID":"dc6f0c46-b1fc-4a23-b93d-f8142042e36d","Type":"ContainerStarted","Data":"28e217ea7bfba558b204bd78ff64ed4e67c6b0a649a2d7a61260cab4e4d293bd"}
Jul 17 00:54:09.963228 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:09.963150    2573 scope.go:117] "RemoveContainer" containerID="e85901f35f2ddcca63b1a152b93ed24c390b7d1b5fc4ff01382a09e662dcca6a"
Jul 17 00:54:10.090920 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.090889    2573 generic.go:356] "Generic (PLEG): container finished" podID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerID="c7b63215d8ab82b790ef862babb4748306db8583da765537e8b61a6402fa721f" exitCode=6
Jul 17 00:54:10.091030 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.090975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" event={"ID":"dc6f0c46-b1fc-4a23-b93d-f8142042e36d","Type":"ContainerDied","Data":"c7b63215d8ab82b790ef862babb4748306db8583da765537e8b61a6402fa721f"}
Jul 17 00:54:10.091132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.091118    2573 scope.go:117] "RemoveContainer" containerID="c7b63215d8ab82b790ef862babb4748306db8583da765537e8b61a6402fa721f"
Jul 17 00:54:10.442198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.442168    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"]
Jul 17 00:54:10.442547 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.442535    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:10.442591 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.442549    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a839a0f-6a3d-4abd-afa0-c4fb7e647b7a" containerName="revoke-keys"
Jul 17 00:54:10.445467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.445452    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"
Jul 17 00:54:10.453079 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.453055    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"]
Jul 17 00:54:10.513091 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.513070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98517bb-67cc-464b-bb44-28fb012efeac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm\" (UID: \"a98517bb-67cc-464b-bb44-28fb012efeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"
Jul 17 00:54:10.613442 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.613388    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98517bb-67cc-464b-bb44-28fb012efeac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm\" (UID: \"a98517bb-67cc-464b-bb44-28fb012efeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"
Jul 17 00:54:10.613966 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.613949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98517bb-67cc-464b-bb44-28fb012efeac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm\" (UID: \"a98517bb-67cc-464b-bb44-28fb012efeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"
Jul 17 00:54:10.757453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.757431    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"
Jul 17 00:54:10.884873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:10.884846    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"]
Jul 17 00:54:10.886056 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:54:10.886028    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98517bb_67cc_464b_bb44_28fb012efeac.slice/crio-38e96597efd3c5a0d1622b1774ba12c84fe63734f1b879a21aead856ee69c47c WatchSource:0}: Error finding container 38e96597efd3c5a0d1622b1774ba12c84fe63734f1b879a21aead856ee69c47c: Status 404 returned error can't find the container with id 38e96597efd3c5a0d1622b1774ba12c84fe63734f1b879a21aead856ee69c47c
Jul 17 00:54:11.098105 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.098080    2573 generic.go:356] "Generic (PLEG): container finished" podID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerID="838aee16be473750970356eec2aca4b689109104251eda7960e358e7bddaf73c" exitCode=7
Jul 17 00:54:11.098406 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.098149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" event={"ID":"276ab6b5-f9c9-4567-8e86-47ecec158461","Type":"ContainerDied","Data":"838aee16be473750970356eec2aca4b689109104251eda7960e358e7bddaf73c"}
Jul 17 00:54:11.098406 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.098204    2573 scope.go:117] "RemoveContainer" containerID="e85901f35f2ddcca63b1a152b93ed24c390b7d1b5fc4ff01382a09e662dcca6a"
Jul 17 00:54:11.098507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.098481    2573 scope.go:117] "RemoveContainer" containerID="838aee16be473750970356eec2aca4b689109104251eda7960e358e7bddaf73c"
Jul 17 00:54:11.098734 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:11.098710    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5p7vf_odh-ai-gateway-infra(276ab6b5-f9c9-4567-8e86-47ecec158461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461"
Jul 17 00:54:11.099897 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.099875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" event={"ID":"a98517bb-67cc-464b-bb44-28fb012efeac","Type":"ContainerStarted","Data":"9b7628808ed6f6efa187892f1fb41b780a5ac436d49a0334a8ea4e8e5c048cc7"}
Jul 17 00:54:11.099979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.099928    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" event={"ID":"a98517bb-67cc-464b-bb44-28fb012efeac","Type":"ContainerStarted","Data":"38e96597efd3c5a0d1622b1774ba12c84fe63734f1b879a21aead856ee69c47c"}
Jul 17 00:54:11.101659 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.101636    2573 generic.go:356] "Generic (PLEG): container finished" podID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerID="bbe31c0c5afae996efc43aebda3b75dc9f12fcb7c0e1090909a124e25c714954" exitCode=6
Jul 17 00:54:11.101757 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.101693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" event={"ID":"dc6f0c46-b1fc-4a23-b93d-f8142042e36d","Type":"ContainerDied","Data":"bbe31c0c5afae996efc43aebda3b75dc9f12fcb7c0e1090909a124e25c714954"}
Jul 17 00:54:11.101899 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.101879    2573 scope.go:117] "RemoveContainer" containerID="bbe31c0c5afae996efc43aebda3b75dc9f12fcb7c0e1090909a124e25c714954"
Jul 17 00:54:11.102035 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:11.102021    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c_odh-ai-gateway-infra(dc6f0c46-b1fc-4a23-b93d-f8142042e36d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d"
Jul 17 00:54:11.108714 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.108698    2573 scope.go:117] "RemoveContainer" containerID="c7b63215d8ab82b790ef862babb4748306db8583da765537e8b61a6402fa721f"
Jul 17 00:54:11.142625 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:11.142561    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" podStartSLOduration=1.142548472 podStartE2EDuration="1.142548472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:54:11.1407111 +0000 UTC m=+2245.716863675" watchObservedRunningTime="2026-07-17 00:54:11.142548472 +0000 UTC m=+2245.718701045"
Jul 17 00:54:12.108667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.108637    2573 generic.go:356] "Generic (PLEG): container finished" podID="a98517bb-67cc-464b-bb44-28fb012efeac" containerID="9b7628808ed6f6efa187892f1fb41b780a5ac436d49a0334a8ea4e8e5c048cc7" exitCode=6
Jul 17 00:54:12.109097 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.108724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" event={"ID":"a98517bb-67cc-464b-bb44-28fb012efeac","Type":"ContainerDied","Data":"9b7628808ed6f6efa187892f1fb41b780a5ac436d49a0334a8ea4e8e5c048cc7"}
Jul 17 00:54:12.109097 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.108878    2573 scope.go:117] "RemoveContainer" containerID="9b7628808ed6f6efa187892f1fb41b780a5ac436d49a0334a8ea4e8e5c048cc7"
Jul 17 00:54:12.110507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.110490    2573 scope.go:117] "RemoveContainer" containerID="bbe31c0c5afae996efc43aebda3b75dc9f12fcb7c0e1090909a124e25c714954"
Jul 17 00:54:12.110645 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:12.110624    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c_odh-ai-gateway-infra(dc6f0c46-b1fc-4a23-b93d-f8142042e36d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d"
Jul 17 00:54:12.124037 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.124010    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"]
Jul 17 00:54:12.265132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.265109    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"
Jul 17 00:54:12.428050 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.428028    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276ab6b5-f9c9-4567-8e86-47ecec158461-maas-api-service-ca\") pod \"276ab6b5-f9c9-4567-8e86-47ecec158461\" (UID: \"276ab6b5-f9c9-4567-8e86-47ecec158461\") "
Jul 17 00:54:12.428343 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.428323    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276ab6b5-f9c9-4567-8e86-47ecec158461-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "276ab6b5-f9c9-4567-8e86-47ecec158461" (UID: "276ab6b5-f9c9-4567-8e86-47ecec158461"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:54:12.529417 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:12.529394    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276ab6b5-f9c9-4567-8e86-47ecec158461-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:54:13.121983 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.121954    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"
Jul 17 00:54:13.122384 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.121995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf" event={"ID":"276ab6b5-f9c9-4567-8e86-47ecec158461","Type":"ContainerDied","Data":"f4b0e48c34e73bbb8306dd57d7bcd9ae9985794c04de1d741c319d620a4d5625"}
Jul 17 00:54:13.122384 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.122055    2573 scope.go:117] "RemoveContainer" containerID="838aee16be473750970356eec2aca4b689109104251eda7960e358e7bddaf73c"
Jul 17 00:54:13.124226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.124198    2573 generic.go:356] "Generic (PLEG): container finished" podID="a98517bb-67cc-464b-bb44-28fb012efeac" containerID="cd8823c5865ba0170b8bf7ad8f961954bcb09e8277314c8b2c1dc493063b5da2" exitCode=6
Jul 17 00:54:13.124345 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.124274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" event={"ID":"a98517bb-67cc-464b-bb44-28fb012efeac","Type":"ContainerDied","Data":"cd8823c5865ba0170b8bf7ad8f961954bcb09e8277314c8b2c1dc493063b5da2"}
Jul 17 00:54:13.124479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.124465    2573 scope.go:117] "RemoveContainer" containerID="cd8823c5865ba0170b8bf7ad8f961954bcb09e8277314c8b2c1dc493063b5da2"
Jul 17 00:54:13.124661 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:13.124643    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm_odh-ai-gateway-infra(a98517bb-67cc-464b-bb44-28fb012efeac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" podUID="a98517bb-67cc-464b-bb44-28fb012efeac"
Jul 17 00:54:13.131603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.131576    2573 scope.go:117] "RemoveContainer" containerID="9b7628808ed6f6efa187892f1fb41b780a5ac436d49a0334a8ea4e8e5c048cc7"
Jul 17 00:54:13.149712 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.149692    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"]
Jul 17 00:54:13.154825 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.154805    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5p7vf"]
Jul 17 00:54:13.974895 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:13.974864    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" path="/var/lib/kubelet/pods/276ab6b5-f9c9-4567-8e86-47ecec158461/volumes"
Jul 17 00:54:14.130729 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:14.130707    2573 scope.go:117] "RemoveContainer" containerID="cd8823c5865ba0170b8bf7ad8f961954bcb09e8277314c8b2c1dc493063b5da2"
Jul 17 00:54:14.131196 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:14.130886    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm_odh-ai-gateway-infra(a98517bb-67cc-464b-bb44-28fb012efeac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" podUID="a98517bb-67cc-464b-bb44-28fb012efeac"
Jul 17 00:54:16.962575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:16.962548    2573 scope.go:117] "RemoveContainer" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"
Jul 17 00:54:16.962988 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:16.962701    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:54:23.962575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:23.962543    2573 scope.go:117] "RemoveContainer" containerID="bbe31c0c5afae996efc43aebda3b75dc9f12fcb7c0e1090909a124e25c714954"
Jul 17 00:54:24.191114 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:24.191086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" event={"ID":"dc6f0c46-b1fc-4a23-b93d-f8142042e36d","Type":"ContainerStarted","Data":"6b55fdc3934bd1dbd125de2b4010194b308bdad6e138619a4c6ed1781a6b2bd3"}
Jul 17 00:54:24.215840 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:24.215746    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" podStartSLOduration=16.215732037 podStartE2EDuration="16.215732037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:54:24.213170708 +0000 UTC m=+2258.789323283" watchObservedRunningTime="2026-07-17 00:54:24.215732037 +0000 UTC m=+2258.791884612"
Jul 17 00:54:25.198330 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:25.198296    2573 generic.go:356] "Generic (PLEG): container finished" podID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerID="6b55fdc3934bd1dbd125de2b4010194b308bdad6e138619a4c6ed1781a6b2bd3" exitCode=6
Jul 17 00:54:25.198732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:25.198344    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" event={"ID":"dc6f0c46-b1fc-4a23-b93d-f8142042e36d","Type":"ContainerDied","Data":"6b55fdc3934bd1dbd125de2b4010194b308bdad6e138619a4c6ed1781a6b2bd3"}
Jul 17 00:54:25.198732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:25.198395    2573 scope.go:117] "RemoveContainer" containerID="bbe31c0c5afae996efc43aebda3b75dc9f12fcb7c0e1090909a124e25c714954"
Jul 17 00:54:25.198732 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:25.198604    2573 scope.go:117] "RemoveContainer" containerID="6b55fdc3934bd1dbd125de2b4010194b308bdad6e138619a4c6ed1781a6b2bd3"
Jul 17 00:54:25.198903 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:25.198839    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c_odh-ai-gateway-infra(dc6f0c46-b1fc-4a23-b93d-f8142042e36d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d"
Jul 17 00:54:25.223000 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:25.222975    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"]
Jul 17 00:54:26.339858 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:26.339834    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"
Jul 17 00:54:26.433287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:26.433257    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f0c46-b1fc-4a23-b93d-f8142042e36d-maas-api-service-ca\") pod \"dc6f0c46-b1fc-4a23-b93d-f8142042e36d\" (UID: \"dc6f0c46-b1fc-4a23-b93d-f8142042e36d\") "
Jul 17 00:54:26.433641 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:26.433620    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6f0c46-b1fc-4a23-b93d-f8142042e36d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc6f0c46-b1fc-4a23-b93d-f8142042e36d" (UID: "dc6f0c46-b1fc-4a23-b93d-f8142042e36d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:54:26.539926 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:26.534762    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6f0c46-b1fc-4a23-b93d-f8142042e36d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:54:26.962982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:26.962907    2573 scope.go:117] "RemoveContainer" containerID="cd8823c5865ba0170b8bf7ad8f961954bcb09e8277314c8b2c1dc493063b5da2"
Jul 17 00:54:27.212857 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:27.212827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c" event={"ID":"dc6f0c46-b1fc-4a23-b93d-f8142042e36d","Type":"ContainerDied","Data":"28e217ea7bfba558b204bd78ff64ed4e67c6b0a649a2d7a61260cab4e4d293bd"}
Jul 17 00:54:27.212965 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:27.212900    2573 scope.go:117] "RemoveContainer" containerID="6b55fdc3934bd1dbd125de2b4010194b308bdad6e138619a4c6ed1781a6b2bd3"
Jul 17 00:54:27.212965 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:27.212845    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"
Jul 17 00:54:27.215085 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:27.215056    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" event={"ID":"a98517bb-67cc-464b-bb44-28fb012efeac","Type":"ContainerStarted","Data":"fd9b4b5cc649ab882e5a14e2ee4e6f0fac0bb60a58435bafb4759da0e1b89934"}
Jul 17 00:54:27.239445 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:27.239424    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"]
Jul 17 00:54:27.242391 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:27.242371    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-q8p2c"]
Jul 17 00:54:27.969077 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:27.969016    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" path="/var/lib/kubelet/pods/dc6f0c46-b1fc-4a23-b93d-f8142042e36d/volumes"
Jul 17 00:54:28.222370 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:28.222313    2573 generic.go:356] "Generic (PLEG): container finished" podID="a98517bb-67cc-464b-bb44-28fb012efeac" containerID="fd9b4b5cc649ab882e5a14e2ee4e6f0fac0bb60a58435bafb4759da0e1b89934" exitCode=6
Jul 17 00:54:28.222511 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:28.222385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" event={"ID":"a98517bb-67cc-464b-bb44-28fb012efeac","Type":"ContainerDied","Data":"fd9b4b5cc649ab882e5a14e2ee4e6f0fac0bb60a58435bafb4759da0e1b89934"}
Jul 17 00:54:28.222511 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:28.222436    2573 scope.go:117] "RemoveContainer" containerID="cd8823c5865ba0170b8bf7ad8f961954bcb09e8277314c8b2c1dc493063b5da2"
Jul 17 00:54:28.222675 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:28.222657    2573 scope.go:117] "RemoveContainer" containerID="fd9b4b5cc649ab882e5a14e2ee4e6f0fac0bb60a58435bafb4759da0e1b89934"
Jul 17 00:54:28.222903 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:28.222879    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm_odh-ai-gateway-infra(a98517bb-67cc-464b-bb44-28fb012efeac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" podUID="a98517bb-67cc-464b-bb44-28fb012efeac"
Jul 17 00:54:28.237877 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:28.237855    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"]
Jul 17 00:54:29.368447 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:29.368421    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"
Jul 17 00:54:29.455306 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:29.455279    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98517bb-67cc-464b-bb44-28fb012efeac-maas-api-service-ca\") pod \"a98517bb-67cc-464b-bb44-28fb012efeac\" (UID: \"a98517bb-67cc-464b-bb44-28fb012efeac\") "
Jul 17 00:54:29.455590 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:29.455570    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98517bb-67cc-464b-bb44-28fb012efeac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a98517bb-67cc-464b-bb44-28fb012efeac" (UID: "a98517bb-67cc-464b-bb44-28fb012efeac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:54:29.556128 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:29.556106    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98517bb-67cc-464b-bb44-28fb012efeac-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:54:29.962977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:29.962897    2573 scope.go:117] "RemoveContainer" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"
Jul 17 00:54:30.237659 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:30.237579    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"
Jul 17 00:54:30.237659 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:30.237594    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm" event={"ID":"a98517bb-67cc-464b-bb44-28fb012efeac","Type":"ContainerDied","Data":"38e96597efd3c5a0d1622b1774ba12c84fe63734f1b879a21aead856ee69c47c"}
Jul 17 00:54:30.237920 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:30.237664    2573 scope.go:117] "RemoveContainer" containerID="fd9b4b5cc649ab882e5a14e2ee4e6f0fac0bb60a58435bafb4759da0e1b89934"
Jul 17 00:54:30.252609 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:30.252582    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"]
Jul 17 00:54:30.257533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:30.257511    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-kzmmm"]
Jul 17 00:54:31.250516 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:31.250427    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899" exitCode=1
Jul 17 00:54:31.250516 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:31.250497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerDied","Data":"a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"}
Jul 17 00:54:31.250998 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:31.250578    2573 scope.go:117] "RemoveContainer" containerID="0f35223225c6576accfd0ba53096de719d9388e74b636e7e7b7bef5c738bfddd"
Jul 17 00:54:31.250998 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:31.250895    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:54:31.251105 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:31.251085    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:54:31.969200 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:31.969169    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" path="/var/lib/kubelet/pods/a98517bb-67cc-464b-bb44-28fb012efeac/volumes"
Jul 17 00:54:32.419639 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.419611    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:54:32.420046 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.419936    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:54:32.420099 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:32.420076    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:54:32.905734 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.905706    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"]
Jul 17 00:54:32.906088 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906074    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906091    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906099    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906104    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906112    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906118    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906127    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906132    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:32.906144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906145    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906150    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906162    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906167    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906175    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906180    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906232    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906239    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906246    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906253    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906259    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:32.906392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.906264    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:32.910748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.910732    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"
Jul 17 00:54:32.912728 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.912709    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:54:32.915094 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:32.915069    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"]
Jul 17 00:54:33.085367 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:33.085340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed038d-a8e3-4d80-95f2-18d53d077c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6\" (UID: \"6aed038d-a8e3-4d80-95f2-18d53d077c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"
Jul 17 00:54:33.185995 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:33.185933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed038d-a8e3-4d80-95f2-18d53d077c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6\" (UID: \"6aed038d-a8e3-4d80-95f2-18d53d077c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"
Jul 17 00:54:33.186507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:33.186488    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed038d-a8e3-4d80-95f2-18d53d077c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6\" (UID: \"6aed038d-a8e3-4d80-95f2-18d53d077c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"
Jul 17 00:54:33.222405 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:33.222387    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"
Jul 17 00:54:33.346692 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:33.346666    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"]
Jul 17 00:54:33.348595 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:54:33.348566    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aed038d_a8e3_4d80_95f2_18d53d077c16.slice/crio-2a1342353f8955fc4279c923e39d2b7b752d20040d4608047f79bfb7004ae99c WatchSource:0}: Error finding container 2a1342353f8955fc4279c923e39d2b7b752d20040d4608047f79bfb7004ae99c: Status 404 returned error can't find the container with id 2a1342353f8955fc4279c923e39d2b7b752d20040d4608047f79bfb7004ae99c
Jul 17 00:54:33.660466 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:33.660445    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:54:33.660779 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:33.660764    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:54:33.660950 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:33.660932    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:54:34.271440 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:34.271399    2573 generic.go:356] "Generic (PLEG): container finished" podID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerID="962deb8248391af40c4778fefdc17024d6d0d73b603dfea9eb2dea7d2747b50e" exitCode=6
Jul 17 00:54:34.271611 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:34.271465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" event={"ID":"6aed038d-a8e3-4d80-95f2-18d53d077c16","Type":"ContainerDied","Data":"962deb8248391af40c4778fefdc17024d6d0d73b603dfea9eb2dea7d2747b50e"}
Jul 17 00:54:34.271611 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:34.271508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" event={"ID":"6aed038d-a8e3-4d80-95f2-18d53d077c16","Type":"ContainerStarted","Data":"2a1342353f8955fc4279c923e39d2b7b752d20040d4608047f79bfb7004ae99c"}
Jul 17 00:54:34.271707 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:34.271643    2573 scope.go:117] "RemoveContainer" containerID="962deb8248391af40c4778fefdc17024d6d0d73b603dfea9eb2dea7d2747b50e"
Jul 17 00:54:35.278673 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:35.278637    2573 generic.go:356] "Generic (PLEG): container finished" podID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerID="2c215cae26c6f4530919f261fa69b196aa42c5ba4a8f2131eeba4aa0d24cc84e" exitCode=6
Jul 17 00:54:35.279118 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:35.278721    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" event={"ID":"6aed038d-a8e3-4d80-95f2-18d53d077c16","Type":"ContainerDied","Data":"2c215cae26c6f4530919f261fa69b196aa42c5ba4a8f2131eeba4aa0d24cc84e"}
Jul 17 00:54:35.279118 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:35.278820    2573 scope.go:117] "RemoveContainer" containerID="962deb8248391af40c4778fefdc17024d6d0d73b603dfea9eb2dea7d2747b50e"
Jul 17 00:54:35.279118 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:35.278905    2573 scope.go:117] "RemoveContainer" containerID="2c215cae26c6f4530919f261fa69b196aa42c5ba4a8f2131eeba4aa0d24cc84e"
Jul 17 00:54:35.279118 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:35.279087    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6_odh-ai-gateway-infra(6aed038d-a8e3-4d80-95f2-18d53d077c16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16"
Jul 17 00:54:36.286079 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:36.286051    2573 scope.go:117] "RemoveContainer" containerID="2c215cae26c6f4530919f261fa69b196aa42c5ba4a8f2131eeba4aa0d24cc84e"
Jul 17 00:54:36.286476 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:36.286218    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6_odh-ai-gateway-infra(6aed038d-a8e3-4d80-95f2-18d53d077c16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16"
Jul 17 00:54:44.962679 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:44.962641    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:54:44.963296 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:44.962934    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:54:45.926928 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.926895    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"]
Jul 17 00:54:45.927260 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.927247    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:45.927316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.927262    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:45.927316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.927290    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:45.927316 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.927296    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:45.927424 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.927354    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="276ab6b5-f9c9-4567-8e86-47ecec158461" containerName="revoke-keys"
Jul 17 00:54:45.927424 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.927364    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98517bb-67cc-464b-bb44-28fb012efeac" containerName="revoke-keys"
Jul 17 00:54:45.927424 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.927371    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6f0c46-b1fc-4a23-b93d-f8142042e36d" containerName="revoke-keys"
Jul 17 00:54:45.931212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.931193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"
Jul 17 00:54:45.936973 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.936672    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"]
Jul 17 00:54:45.978990 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:45.978959    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de35e4a-c8ee-4789-a894-a9c8cb751049-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-g7wdj\" (UID: \"5de35e4a-c8ee-4789-a894-a9c8cb751049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"
Jul 17 00:54:46.079612 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:46.079581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de35e4a-c8ee-4789-a894-a9c8cb751049-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-g7wdj\" (UID: \"5de35e4a-c8ee-4789-a894-a9c8cb751049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"
Jul 17 00:54:46.080207 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:46.080183    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de35e4a-c8ee-4789-a894-a9c8cb751049-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-g7wdj\" (UID: \"5de35e4a-c8ee-4789-a894-a9c8cb751049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"
Jul 17 00:54:46.243286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:46.243206    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"
Jul 17 00:54:46.372226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:46.372202    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"]
Jul 17 00:54:46.373932 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:54:46.373905    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de35e4a_c8ee_4789_a894_a9c8cb751049.slice/crio-286efa276577c3e4ebfdc03dba838816048dcdbf2ac737f02f3199b685172324 WatchSource:0}: Error finding container 286efa276577c3e4ebfdc03dba838816048dcdbf2ac737f02f3199b685172324: Status 404 returned error can't find the container with id 286efa276577c3e4ebfdc03dba838816048dcdbf2ac737f02f3199b685172324
Jul 17 00:54:47.345969 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:47.345933    2573 generic.go:356] "Generic (PLEG): container finished" podID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerID="9ae009894d9d16981388521285e839283cafd2b8d40a7cd79912e7c9ea5b5d47" exitCode=7
Jul 17 00:54:47.346286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:47.346016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" event={"ID":"5de35e4a-c8ee-4789-a894-a9c8cb751049","Type":"ContainerDied","Data":"9ae009894d9d16981388521285e839283cafd2b8d40a7cd79912e7c9ea5b5d47"}
Jul 17 00:54:47.346286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:47.346092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" event={"ID":"5de35e4a-c8ee-4789-a894-a9c8cb751049","Type":"ContainerStarted","Data":"286efa276577c3e4ebfdc03dba838816048dcdbf2ac737f02f3199b685172324"}
Jul 17 00:54:47.346286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:47.346152    2573 scope.go:117] "RemoveContainer" containerID="9ae009894d9d16981388521285e839283cafd2b8d40a7cd79912e7c9ea5b5d47"
Jul 17 00:54:48.353135 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:48.353104    2573 generic.go:356] "Generic (PLEG): container finished" podID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerID="f68e1bb789ba98111f394ee887720cd9c90573b2ed8c1fb3662a30a462174656" exitCode=7
Jul 17 00:54:48.353542 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:48.353168    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" event={"ID":"5de35e4a-c8ee-4789-a894-a9c8cb751049","Type":"ContainerDied","Data":"f68e1bb789ba98111f394ee887720cd9c90573b2ed8c1fb3662a30a462174656"}
Jul 17 00:54:48.353542 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:48.353225    2573 scope.go:117] "RemoveContainer" containerID="9ae009894d9d16981388521285e839283cafd2b8d40a7cd79912e7c9ea5b5d47"
Jul 17 00:54:48.353542 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:48.353392    2573 scope.go:117] "RemoveContainer" containerID="f68e1bb789ba98111f394ee887720cd9c90573b2ed8c1fb3662a30a462174656"
Jul 17 00:54:48.353758 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:48.353602    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-g7wdj_odh-ai-gateway-infra(5de35e4a-c8ee-4789-a894-a9c8cb751049)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049"
Jul 17 00:54:49.360198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:49.360167    2573 scope.go:117] "RemoveContainer" containerID="f68e1bb789ba98111f394ee887720cd9c90573b2ed8c1fb3662a30a462174656"
Jul 17 00:54:49.360550 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:49.360326    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-g7wdj_odh-ai-gateway-infra(5de35e4a-c8ee-4789-a894-a9c8cb751049)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049"
Jul 17 00:54:50.962684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:50.962656    2573 scope.go:117] "RemoveContainer" containerID="2c215cae26c6f4530919f261fa69b196aa42c5ba4a8f2131eeba4aa0d24cc84e"
Jul 17 00:54:51.371700 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:51.371676    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" event={"ID":"6aed038d-a8e3-4d80-95f2-18d53d077c16","Type":"ContainerStarted","Data":"2f9e0556a85042118397cd883a323829babc86dcf57671702f13822679e5e2dd"}
Jul 17 00:54:51.386171 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:51.386131    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" podStartSLOduration=19.386116622 podStartE2EDuration="19.386116622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:54:51.384255381 +0000 UTC m=+2285.960407964" watchObservedRunningTime="2026-07-17 00:54:51.386116622 +0000 UTC m=+2285.962269209"
Jul 17 00:54:52.378282 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:52.378248    2573 generic.go:356] "Generic (PLEG): container finished" podID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerID="2f9e0556a85042118397cd883a323829babc86dcf57671702f13822679e5e2dd" exitCode=6
Jul 17 00:54:52.378695 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:52.378319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" event={"ID":"6aed038d-a8e3-4d80-95f2-18d53d077c16","Type":"ContainerDied","Data":"2f9e0556a85042118397cd883a323829babc86dcf57671702f13822679e5e2dd"}
Jul 17 00:54:52.378695 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:52.378374    2573 scope.go:117] "RemoveContainer" containerID="2c215cae26c6f4530919f261fa69b196aa42c5ba4a8f2131eeba4aa0d24cc84e"
Jul 17 00:54:52.378695 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:52.378650    2573 scope.go:117] "RemoveContainer" containerID="2f9e0556a85042118397cd883a323829babc86dcf57671702f13822679e5e2dd"
Jul 17 00:54:52.378838 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:52.378821    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6_odh-ai-gateway-infra(6aed038d-a8e3-4d80-95f2-18d53d077c16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16"
Jul 17 00:54:52.403009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:52.402984    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"]
Jul 17 00:54:53.523246 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:53.523218    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"
Jul 17 00:54:53.539209 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:53.538680    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed038d-a8e3-4d80-95f2-18d53d077c16-maas-api-service-ca\") pod \"6aed038d-a8e3-4d80-95f2-18d53d077c16\" (UID: \"6aed038d-a8e3-4d80-95f2-18d53d077c16\") "
Jul 17 00:54:53.539348 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:53.539282    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aed038d-a8e3-4d80-95f2-18d53d077c16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6aed038d-a8e3-4d80-95f2-18d53d077c16" (UID: "6aed038d-a8e3-4d80-95f2-18d53d077c16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:54:53.640112 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:53.640039    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed038d-a8e3-4d80-95f2-18d53d077c16-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:54:54.393786 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:54.393752    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"
Jul 17 00:54:54.393971 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:54.393783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6" event={"ID":"6aed038d-a8e3-4d80-95f2-18d53d077c16","Type":"ContainerDied","Data":"2a1342353f8955fc4279c923e39d2b7b752d20040d4608047f79bfb7004ae99c"}
Jul 17 00:54:54.393971 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:54.393856    2573 scope.go:117] "RemoveContainer" containerID="2f9e0556a85042118397cd883a323829babc86dcf57671702f13822679e5e2dd"
Jul 17 00:54:54.408278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:54.408257    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"]
Jul 17 00:54:54.413093 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:54.413075    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-76zg6"]
Jul 17 00:54:55.965999 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:55.965971    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:54:55.966363 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:54:55.966146    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:54:55.968617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:55.968597    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" path="/var/lib/kubelet/pods/6aed038d-a8e3-4d80-95f2-18d53d077c16/volumes"
Jul 17 00:54:58.651902 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.651871    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"]
Jul 17 00:54:58.652294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652228    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.652294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652247    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.652294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652255    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.652294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652261    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.652294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652275    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.652294 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652283    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.652492 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652357    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.652492 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.652372    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:54:58.656863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.656847    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"
Jul 17 00:54:58.661192 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.661167    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"]
Jul 17 00:54:58.684896 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.684874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f3433-de87-4ba9-8967-5949ff3e946a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r\" (UID: \"c57f3433-de87-4ba9-8967-5949ff3e946a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"
Jul 17 00:54:58.786061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.786037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f3433-de87-4ba9-8967-5949ff3e946a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r\" (UID: \"c57f3433-de87-4ba9-8967-5949ff3e946a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"
Jul 17 00:54:58.786601 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.786583    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f3433-de87-4ba9-8967-5949ff3e946a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r\" (UID: \"c57f3433-de87-4ba9-8967-5949ff3e946a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"
Jul 17 00:54:58.969424 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:58.969366    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"
Jul 17 00:54:59.097381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:59.097359    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"]
Jul 17 00:54:59.099831 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:54:59.099782    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57f3433_de87_4ba9_8967_5949ff3e946a.slice/crio-352960cf6ae014d310821705cb45383b6c9b43fa01de8d05fb8a22c265f405ff WatchSource:0}: Error finding container 352960cf6ae014d310821705cb45383b6c9b43fa01de8d05fb8a22c265f405ff: Status 404 returned error can't find the container with id 352960cf6ae014d310821705cb45383b6c9b43fa01de8d05fb8a22c265f405ff
Jul 17 00:54:59.421701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:59.421669    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" event={"ID":"c57f3433-de87-4ba9-8967-5949ff3e946a","Type":"ContainerStarted","Data":"48b95840791ba82f0d2e68fd465829c4494230a49ee93548ba5a54f360d04a25"}
Jul 17 00:54:59.421835 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:59.421731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" event={"ID":"c57f3433-de87-4ba9-8967-5949ff3e946a","Type":"ContainerStarted","Data":"352960cf6ae014d310821705cb45383b6c9b43fa01de8d05fb8a22c265f405ff"}
Jul 17 00:54:59.438324 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:54:59.438282    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" podStartSLOduration=1.43826574 podStartE2EDuration="1.43826574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:54:59.436024763 +0000 UTC m=+2294.012177347" watchObservedRunningTime="2026-07-17 00:54:59.43826574 +0000 UTC m=+2294.014418351"
Jul 17 00:55:00.428521 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.428489    2573 generic.go:356] "Generic (PLEG): container finished" podID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerID="48b95840791ba82f0d2e68fd465829c4494230a49ee93548ba5a54f360d04a25" exitCode=6
Jul 17 00:55:00.428917 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.428567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" event={"ID":"c57f3433-de87-4ba9-8967-5949ff3e946a","Type":"ContainerDied","Data":"48b95840791ba82f0d2e68fd465829c4494230a49ee93548ba5a54f360d04a25"}
Jul 17 00:55:00.428917 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.428764    2573 scope.go:117] "RemoveContainer" containerID="48b95840791ba82f0d2e68fd465829c4494230a49ee93548ba5a54f360d04a25"
Jul 17 00:55:00.481708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.481676    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"]
Jul 17 00:55:00.482127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.482113    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aed038d-a8e3-4d80-95f2-18d53d077c16" containerName="revoke-keys"
Jul 17 00:55:00.486500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.486485    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"
Jul 17 00:55:00.491525 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.491502    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"]
Jul 17 00:55:00.602909 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.602880    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957da187-7352-4327-89b7-787323d28a81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm\" (UID: \"957da187-7352-4327-89b7-787323d28a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"
Jul 17 00:55:00.704206 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.704141    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957da187-7352-4327-89b7-787323d28a81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm\" (UID: \"957da187-7352-4327-89b7-787323d28a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"
Jul 17 00:55:00.704731 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.704710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957da187-7352-4327-89b7-787323d28a81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm\" (UID: \"957da187-7352-4327-89b7-787323d28a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"
Jul 17 00:55:00.797169 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.797139    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"
Jul 17 00:55:00.918568 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:00.918544    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"]
Jul 17 00:55:00.920649 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:55:00.920618    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957da187_7352_4327_89b7_787323d28a81.slice/crio-87a5c3b47c7976c81a80507f308417e571936eb12720cea62bf200d5c521b69c WatchSource:0}: Error finding container 87a5c3b47c7976c81a80507f308417e571936eb12720cea62bf200d5c521b69c: Status 404 returned error can't find the container with id 87a5c3b47c7976c81a80507f308417e571936eb12720cea62bf200d5c521b69c
Jul 17 00:55:01.435221 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.435192    2573 generic.go:356] "Generic (PLEG): container finished" podID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerID="241f8a657d2067f792315c4168162aeedb026d2c471b7f4b2b55fb2ed0e8c148" exitCode=6
Jul 17 00:55:01.435635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.435258    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" event={"ID":"c57f3433-de87-4ba9-8967-5949ff3e946a","Type":"ContainerDied","Data":"241f8a657d2067f792315c4168162aeedb026d2c471b7f4b2b55fb2ed0e8c148"}
Jul 17 00:55:01.435635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.435307    2573 scope.go:117] "RemoveContainer" containerID="48b95840791ba82f0d2e68fd465829c4494230a49ee93548ba5a54f360d04a25"
Jul 17 00:55:01.435635 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.435482    2573 scope.go:117] "RemoveContainer" containerID="241f8a657d2067f792315c4168162aeedb026d2c471b7f4b2b55fb2ed0e8c148"
Jul 17 00:55:01.435806 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:01.435700    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r_odh-ai-gateway-infra(c57f3433-de87-4ba9-8967-5949ff3e946a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a"
Jul 17 00:55:01.436841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.436815    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" event={"ID":"957da187-7352-4327-89b7-787323d28a81","Type":"ContainerStarted","Data":"b11f301d2dacb90e0a66c73c9c5ca1dab8cd861b1bbc80237231d4045fc47936"}
Jul 17 00:55:01.436957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.436867    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" event={"ID":"957da187-7352-4327-89b7-787323d28a81","Type":"ContainerStarted","Data":"87a5c3b47c7976c81a80507f308417e571936eb12720cea62bf200d5c521b69c"}
Jul 17 00:55:01.462194 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.462150    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" podStartSLOduration=1.462137913 podStartE2EDuration="1.462137913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:55:01.460541098 +0000 UTC m=+2296.036693671" watchObservedRunningTime="2026-07-17 00:55:01.462137913 +0000 UTC m=+2296.038290506"
Jul 17 00:55:01.962523 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:01.962495    2573 scope.go:117] "RemoveContainer" containerID="f68e1bb789ba98111f394ee887720cd9c90573b2ed8c1fb3662a30a462174656"
Jul 17 00:55:02.444105 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:02.444067    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" event={"ID":"5de35e4a-c8ee-4789-a894-a9c8cb751049","Type":"ContainerStarted","Data":"fc1d107548f151378e1bdeba9acc525f61ce5c86927ad9b50268bab60a4e7640"}
Jul 17 00:55:02.445716 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:02.445690    2573 scope.go:117] "RemoveContainer" containerID="241f8a657d2067f792315c4168162aeedb026d2c471b7f4b2b55fb2ed0e8c148"
Jul 17 00:55:02.445921 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:02.445901    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r_odh-ai-gateway-infra(c57f3433-de87-4ba9-8967-5949ff3e946a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a"
Jul 17 00:55:02.446945 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:02.446912    2573 generic.go:356] "Generic (PLEG): container finished" podID="957da187-7352-4327-89b7-787323d28a81" containerID="b11f301d2dacb90e0a66c73c9c5ca1dab8cd861b1bbc80237231d4045fc47936" exitCode=6
Jul 17 00:55:02.447068 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:02.446947    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" event={"ID":"957da187-7352-4327-89b7-787323d28a81","Type":"ContainerDied","Data":"b11f301d2dacb90e0a66c73c9c5ca1dab8cd861b1bbc80237231d4045fc47936"}
Jul 17 00:55:02.447149 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:02.447139    2573 scope.go:117] "RemoveContainer" containerID="b11f301d2dacb90e0a66c73c9c5ca1dab8cd861b1bbc80237231d4045fc47936"
Jul 17 00:55:02.459006 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:02.458960    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" podStartSLOduration=17.458949114 podStartE2EDuration="17.458949114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:55:02.457108711 +0000 UTC m=+2297.033261285" watchObservedRunningTime="2026-07-17 00:55:02.458949114 +0000 UTC m=+2297.035101687"
Jul 17 00:55:03.454118 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.454024    2573 generic.go:356] "Generic (PLEG): container finished" podID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerID="fc1d107548f151378e1bdeba9acc525f61ce5c86927ad9b50268bab60a4e7640" exitCode=7
Jul 17 00:55:03.454118 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.454095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" event={"ID":"5de35e4a-c8ee-4789-a894-a9c8cb751049","Type":"ContainerDied","Data":"fc1d107548f151378e1bdeba9acc525f61ce5c86927ad9b50268bab60a4e7640"}
Jul 17 00:55:03.454583 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.454156    2573 scope.go:117] "RemoveContainer" containerID="f68e1bb789ba98111f394ee887720cd9c90573b2ed8c1fb3662a30a462174656"
Jul 17 00:55:03.454583 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.454431    2573 scope.go:117] "RemoveContainer" containerID="fc1d107548f151378e1bdeba9acc525f61ce5c86927ad9b50268bab60a4e7640"
Jul 17 00:55:03.454681 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:03.454661    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-g7wdj_odh-ai-gateway-infra(5de35e4a-c8ee-4789-a894-a9c8cb751049)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049"
Jul 17 00:55:03.455938 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.455917    2573 generic.go:356] "Generic (PLEG): container finished" podID="957da187-7352-4327-89b7-787323d28a81" containerID="5a5c69797bffe71e470a2992f582e4a0be6237607a30f2d61d7576b198d3012e" exitCode=6
Jul 17 00:55:03.456040 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.455945    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" event={"ID":"957da187-7352-4327-89b7-787323d28a81","Type":"ContainerDied","Data":"5a5c69797bffe71e470a2992f582e4a0be6237607a30f2d61d7576b198d3012e"}
Jul 17 00:55:03.456165 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.456150    2573 scope.go:117] "RemoveContainer" containerID="5a5c69797bffe71e470a2992f582e4a0be6237607a30f2d61d7576b198d3012e"
Jul 17 00:55:03.456341 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:03.456323    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm_odh-ai-gateway-infra(957da187-7352-4327-89b7-787323d28a81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" podUID="957da187-7352-4327-89b7-787323d28a81"
Jul 17 00:55:03.465254 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.465229    2573 scope.go:117] "RemoveContainer" containerID="b11f301d2dacb90e0a66c73c9c5ca1dab8cd861b1bbc80237231d4045fc47936"
Jul 17 00:55:03.468013 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:03.467995    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"]
Jul 17 00:55:04.462925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:04.462899    2573 scope.go:117] "RemoveContainer" containerID="5a5c69797bffe71e470a2992f582e4a0be6237607a30f2d61d7576b198d3012e"
Jul 17 00:55:04.463338 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:04.463112    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm_odh-ai-gateway-infra(957da187-7352-4327-89b7-787323d28a81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" podUID="957da187-7352-4327-89b7-787323d28a81"
Jul 17 00:55:04.600580 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:04.600547    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"
Jul 17 00:55:04.738607 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:04.738514    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de35e4a-c8ee-4789-a894-a9c8cb751049-maas-api-service-ca\") pod \"5de35e4a-c8ee-4789-a894-a9c8cb751049\" (UID: \"5de35e4a-c8ee-4789-a894-a9c8cb751049\") "
Jul 17 00:55:04.738901 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:04.738879    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de35e4a-c8ee-4789-a894-a9c8cb751049-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5de35e4a-c8ee-4789-a894-a9c8cb751049" (UID: "5de35e4a-c8ee-4789-a894-a9c8cb751049"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:55:04.839132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:04.839094    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de35e4a-c8ee-4789-a894-a9c8cb751049-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:55:05.471852 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:05.471824    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"
Jul 17 00:55:05.471852 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:05.471839    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj" event={"ID":"5de35e4a-c8ee-4789-a894-a9c8cb751049","Type":"ContainerDied","Data":"286efa276577c3e4ebfdc03dba838816048dcdbf2ac737f02f3199b685172324"}
Jul 17 00:55:05.472307 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:05.471896    2573 scope.go:117] "RemoveContainer" containerID="fc1d107548f151378e1bdeba9acc525f61ce5c86927ad9b50268bab60a4e7640"
Jul 17 00:55:05.491455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:05.491428    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"]
Jul 17 00:55:05.497130 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:05.495477    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g7wdj"]
Jul 17 00:55:05.968417 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:05.968387    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" path="/var/lib/kubelet/pods/5de35e4a-c8ee-4789-a894-a9c8cb751049/volumes"
Jul 17 00:55:09.962300 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:09.962272    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:55:09.962647 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:09.962431    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:55:15.965492 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:15.965460    2573 scope.go:117] "RemoveContainer" containerID="5a5c69797bffe71e470a2992f582e4a0be6237607a30f2d61d7576b198d3012e"
Jul 17 00:55:16.529084 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:16.529052    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" event={"ID":"957da187-7352-4327-89b7-787323d28a81","Type":"ContainerStarted","Data":"90d57964969d722a8192fe6b613271f7ea68c79a3095fa2772de49f9abaacb7f"}
Jul 17 00:55:16.963132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:16.963047    2573 scope.go:117] "RemoveContainer" containerID="241f8a657d2067f792315c4168162aeedb026d2c471b7f4b2b55fb2ed0e8c148"
Jul 17 00:55:17.536340 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:17.536303    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" event={"ID":"c57f3433-de87-4ba9-8967-5949ff3e946a","Type":"ContainerStarted","Data":"d16d7fefafc6dd20844efd087d0eaeb0678524eb57013310fa67f25d2b811e0c"}
Jul 17 00:55:17.538189 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:17.538166    2573 generic.go:356] "Generic (PLEG): container finished" podID="957da187-7352-4327-89b7-787323d28a81" containerID="90d57964969d722a8192fe6b613271f7ea68c79a3095fa2772de49f9abaacb7f" exitCode=6
Jul 17 00:55:17.538318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:17.538249    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" event={"ID":"957da187-7352-4327-89b7-787323d28a81","Type":"ContainerDied","Data":"90d57964969d722a8192fe6b613271f7ea68c79a3095fa2772de49f9abaacb7f"}
Jul 17 00:55:17.538318 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:17.538308    2573 scope.go:117] "RemoveContainer" containerID="5a5c69797bffe71e470a2992f582e4a0be6237607a30f2d61d7576b198d3012e"
Jul 17 00:55:17.538544 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:17.538530    2573 scope.go:117] "RemoveContainer" containerID="90d57964969d722a8192fe6b613271f7ea68c79a3095fa2772de49f9abaacb7f"
Jul 17 00:55:17.538732 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:17.538715    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm_odh-ai-gateway-infra(957da187-7352-4327-89b7-787323d28a81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" podUID="957da187-7352-4327-89b7-787323d28a81"
Jul 17 00:55:17.551454 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:17.551428    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"]
Jul 17 00:55:18.544813 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.544762    2573 generic.go:356] "Generic (PLEG): container finished" podID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerID="d16d7fefafc6dd20844efd087d0eaeb0678524eb57013310fa67f25d2b811e0c" exitCode=6
Jul 17 00:55:18.545274 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.544828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" event={"ID":"c57f3433-de87-4ba9-8967-5949ff3e946a","Type":"ContainerDied","Data":"d16d7fefafc6dd20844efd087d0eaeb0678524eb57013310fa67f25d2b811e0c"}
Jul 17 00:55:18.545274 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.544892    2573 scope.go:117] "RemoveContainer" containerID="241f8a657d2067f792315c4168162aeedb026d2c471b7f4b2b55fb2ed0e8c148"
Jul 17 00:55:18.545274 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.545177    2573 scope.go:117] "RemoveContainer" containerID="d16d7fefafc6dd20844efd087d0eaeb0678524eb57013310fa67f25d2b811e0c"
Jul 17 00:55:18.545440 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:18.545382    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r_odh-ai-gateway-infra(c57f3433-de87-4ba9-8967-5949ff3e946a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a"
Jul 17 00:55:18.560391 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.560367    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"]
Jul 17 00:55:18.687964 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.687943    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"
Jul 17 00:55:18.849183 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.849120    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957da187-7352-4327-89b7-787323d28a81-maas-api-service-ca\") pod \"957da187-7352-4327-89b7-787323d28a81\" (UID: \"957da187-7352-4327-89b7-787323d28a81\") "
Jul 17 00:55:18.849434 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.849414    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957da187-7352-4327-89b7-787323d28a81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "957da187-7352-4327-89b7-787323d28a81" (UID: "957da187-7352-4327-89b7-787323d28a81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:55:18.950286 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:18.950260    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957da187-7352-4327-89b7-787323d28a81-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:55:19.554742 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.554712    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"
Jul 17 00:55:19.554742 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.554735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm" event={"ID":"957da187-7352-4327-89b7-787323d28a81","Type":"ContainerDied","Data":"87a5c3b47c7976c81a80507f308417e571936eb12720cea62bf200d5c521b69c"}
Jul 17 00:55:19.555193 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.554810    2573 scope.go:117] "RemoveContainer" containerID="90d57964969d722a8192fe6b613271f7ea68c79a3095fa2772de49f9abaacb7f"
Jul 17 00:55:19.578681 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.578655    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"]
Jul 17 00:55:19.581975 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.581952    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-4f4rm"]
Jul 17 00:55:19.688935 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.688913    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"
Jul 17 00:55:19.857515 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.857450    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f3433-de87-4ba9-8967-5949ff3e946a-maas-api-service-ca\") pod \"c57f3433-de87-4ba9-8967-5949ff3e946a\" (UID: \"c57f3433-de87-4ba9-8967-5949ff3e946a\") "
Jul 17 00:55:19.857809 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.857769    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57f3433-de87-4ba9-8967-5949ff3e946a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c57f3433-de87-4ba9-8967-5949ff3e946a" (UID: "c57f3433-de87-4ba9-8967-5949ff3e946a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:55:19.958860 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.958835    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f3433-de87-4ba9-8967-5949ff3e946a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:55:19.968759 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:19.968734    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957da187-7352-4327-89b7-787323d28a81" path="/var/lib/kubelet/pods/957da187-7352-4327-89b7-787323d28a81/volumes"
Jul 17 00:55:20.561307 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:20.561284    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"
Jul 17 00:55:20.561735 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:20.561284    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r" event={"ID":"c57f3433-de87-4ba9-8967-5949ff3e946a","Type":"ContainerDied","Data":"352960cf6ae014d310821705cb45383b6c9b43fa01de8d05fb8a22c265f405ff"}
Jul 17 00:55:20.561735 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:20.561409    2573 scope.go:117] "RemoveContainer" containerID="d16d7fefafc6dd20844efd087d0eaeb0678524eb57013310fa67f25d2b811e0c"
Jul 17 00:55:20.575327 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:20.575306    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"]
Jul 17 00:55:20.578622 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:20.578597    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bxg6r"]
Jul 17 00:55:20.962163 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:20.962096    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:55:20.962280 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:20.962240    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:55:21.968021 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:21.967986    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" path="/var/lib/kubelet/pods/c57f3433-de87-4ba9-8967-5949ff3e946a/volumes"
Jul 17 00:55:32.950603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950569    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"]
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950911    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950922    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950930    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950935    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950944    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950949    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950956    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950960    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950965    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.950979 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950970    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950985    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950990    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.950998    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951003    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951051    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951058    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951064    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951069    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951074    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951079    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951085    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="957da187-7352-4327-89b7-787323d28a81" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951091    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:32.951281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.951095    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:32.954048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.954031    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"
Jul 17 00:55:32.955817 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.955780    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:55:32.959729 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:32.959705    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"]
Jul 17 00:55:33.060248 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.060226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfbb015-395d-43e5-99d8-dc9b2c30f677-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s\" (UID: \"cdfbb015-395d-43e5-99d8-dc9b2c30f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"
Jul 17 00:55:33.161091 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.161069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfbb015-395d-43e5-99d8-dc9b2c30f677-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s\" (UID: \"cdfbb015-395d-43e5-99d8-dc9b2c30f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"
Jul 17 00:55:33.161628 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.161612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfbb015-395d-43e5-99d8-dc9b2c30f677-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s\" (UID: \"cdfbb015-395d-43e5-99d8-dc9b2c30f677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"
Jul 17 00:55:33.266347 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.266327    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"
Jul 17 00:55:33.392036 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.392010    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"]
Jul 17 00:55:33.393551 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:55:33.393526    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfbb015_395d_43e5_99d8_dc9b2c30f677.slice/crio-d1f372ec7c42d1dfb348cb36cfebd08b53fab5ae46e5fe5a3f005dc8a4d8c9a7 WatchSource:0}: Error finding container d1f372ec7c42d1dfb348cb36cfebd08b53fab5ae46e5fe5a3f005dc8a4d8c9a7: Status 404 returned error can't find the container with id d1f372ec7c42d1dfb348cb36cfebd08b53fab5ae46e5fe5a3f005dc8a4d8c9a7
Jul 17 00:55:33.634022 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.633995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" event={"ID":"cdfbb015-395d-43e5-99d8-dc9b2c30f677","Type":"ContainerStarted","Data":"24d7d3e8a96e2b880d3d202fcd1c3099645434012c7130520b205c857beb82d8"}
Jul 17 00:55:33.634125 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.634043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" event={"ID":"cdfbb015-395d-43e5-99d8-dc9b2c30f677","Type":"ContainerStarted","Data":"d1f372ec7c42d1dfb348cb36cfebd08b53fab5ae46e5fe5a3f005dc8a4d8c9a7"}
Jul 17 00:55:33.647673 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.647634    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" podStartSLOduration=1.647622835 podStartE2EDuration="1.647622835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:55:33.646221796 +0000 UTC m=+2328.222374368" watchObservedRunningTime="2026-07-17 00:55:33.647622835 +0000 UTC m=+2328.223775408"
Jul 17 00:55:33.962736 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:33.962719    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:55:33.963060 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:33.962910    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:55:34.640310 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:34.640278    2573 generic.go:356] "Generic (PLEG): container finished" podID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerID="24d7d3e8a96e2b880d3d202fcd1c3099645434012c7130520b205c857beb82d8" exitCode=6
Jul 17 00:55:34.640491 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:34.640346    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" event={"ID":"cdfbb015-395d-43e5-99d8-dc9b2c30f677","Type":"ContainerDied","Data":"24d7d3e8a96e2b880d3d202fcd1c3099645434012c7130520b205c857beb82d8"}
Jul 17 00:55:34.640541 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:34.640519    2573 scope.go:117] "RemoveContainer" containerID="24d7d3e8a96e2b880d3d202fcd1c3099645434012c7130520b205c857beb82d8"
Jul 17 00:55:35.654139 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.654110    2573 generic.go:356] "Generic (PLEG): container finished" podID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerID="3c780237cd19b09977f83b55b0ec47412423e3d527783e4ac84f1d3803153b3f" exitCode=6
Jul 17 00:55:35.654574 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.654200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" event={"ID":"cdfbb015-395d-43e5-99d8-dc9b2c30f677","Type":"ContainerDied","Data":"3c780237cd19b09977f83b55b0ec47412423e3d527783e4ac84f1d3803153b3f"}
Jul 17 00:55:35.654574 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.654281    2573 scope.go:117] "RemoveContainer" containerID="24d7d3e8a96e2b880d3d202fcd1c3099645434012c7130520b205c857beb82d8"
Jul 17 00:55:35.654574 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.654474    2573 scope.go:117] "RemoveContainer" containerID="3c780237cd19b09977f83b55b0ec47412423e3d527783e4ac84f1d3803153b3f"
Jul 17 00:55:35.654716 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:35.654637    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s_odh-ai-gateway-infra(cdfbb015-395d-43e5-99d8-dc9b2c30f677)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677"
Jul 17 00:55:35.968256 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.968182    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"]
Jul 17 00:55:35.968588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.968571    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:35.968667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.968590    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57f3433-de87-4ba9-8967-5949ff3e946a" containerName="revoke-keys"
Jul 17 00:55:35.968667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.968610    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:35.968667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.968618    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de35e4a-c8ee-4789-a894-a9c8cb751049" containerName="revoke-keys"
Jul 17 00:55:35.972910 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.972886    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"]
Jul 17 00:55:35.973033 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:35.972978    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"
Jul 17 00:55:36.082245 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:36.082215    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500540df-576a-4499-88b7-0ffdee19a9fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-kvl5l\" (UID: \"500540df-576a-4499-88b7-0ffdee19a9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"
Jul 17 00:55:36.183288 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:36.183255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500540df-576a-4499-88b7-0ffdee19a9fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-kvl5l\" (UID: \"500540df-576a-4499-88b7-0ffdee19a9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"
Jul 17 00:55:36.183851 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:36.183833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500540df-576a-4499-88b7-0ffdee19a9fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-kvl5l\" (UID: \"500540df-576a-4499-88b7-0ffdee19a9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"
Jul 17 00:55:36.284359 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:36.284334    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"
Jul 17 00:55:36.618262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:36.618237    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"]
Jul 17 00:55:36.620139 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:55:36.620110    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500540df_576a_4499_88b7_0ffdee19a9fe.slice/crio-a436d89b6d158c1ac4f1316f597d39aee7a899866c97d706df5a1f3306324bdd WatchSource:0}: Error finding container a436d89b6d158c1ac4f1316f597d39aee7a899866c97d706df5a1f3306324bdd: Status 404 returned error can't find the container with id a436d89b6d158c1ac4f1316f597d39aee7a899866c97d706df5a1f3306324bdd
Jul 17 00:55:36.661387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:36.661371    2573 scope.go:117] "RemoveContainer" containerID="3c780237cd19b09977f83b55b0ec47412423e3d527783e4ac84f1d3803153b3f"
Jul 17 00:55:36.661723 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:36.661600    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s_odh-ai-gateway-infra(cdfbb015-395d-43e5-99d8-dc9b2c30f677)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677"
Jul 17 00:55:36.662619 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:36.662588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" event={"ID":"500540df-576a-4499-88b7-0ffdee19a9fe","Type":"ContainerStarted","Data":"a436d89b6d158c1ac4f1316f597d39aee7a899866c97d706df5a1f3306324bdd"}
Jul 17 00:55:37.669302 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:37.669273    2573 generic.go:356] "Generic (PLEG): container finished" podID="500540df-576a-4499-88b7-0ffdee19a9fe" containerID="72c5d9cf61342a0859ee6960c2305aa34908d0c882e01da961cedc7688e7c2a8" exitCode=7
Jul 17 00:55:37.669677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:37.669358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" event={"ID":"500540df-576a-4499-88b7-0ffdee19a9fe","Type":"ContainerDied","Data":"72c5d9cf61342a0859ee6960c2305aa34908d0c882e01da961cedc7688e7c2a8"}
Jul 17 00:55:37.669677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:37.669470    2573 scope.go:117] "RemoveContainer" containerID="72c5d9cf61342a0859ee6960c2305aa34908d0c882e01da961cedc7688e7c2a8"
Jul 17 00:55:38.676729 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:38.676697    2573 generic.go:356] "Generic (PLEG): container finished" podID="500540df-576a-4499-88b7-0ffdee19a9fe" containerID="0a6835f6b7aae7eb3301a38310d07ed55ab22f773876d526c11d14793019b81b" exitCode=7
Jul 17 00:55:38.677151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:38.676777    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" event={"ID":"500540df-576a-4499-88b7-0ffdee19a9fe","Type":"ContainerDied","Data":"0a6835f6b7aae7eb3301a38310d07ed55ab22f773876d526c11d14793019b81b"}
Jul 17 00:55:38.677151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:38.676850    2573 scope.go:117] "RemoveContainer" containerID="72c5d9cf61342a0859ee6960c2305aa34908d0c882e01da961cedc7688e7c2a8"
Jul 17 00:55:38.677151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:38.677057    2573 scope.go:117] "RemoveContainer" containerID="0a6835f6b7aae7eb3301a38310d07ed55ab22f773876d526c11d14793019b81b"
Jul 17 00:55:38.677265 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:38.677242    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-kvl5l_odh-ai-gateway-infra(500540df-576a-4499-88b7-0ffdee19a9fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" podUID="500540df-576a-4499-88b7-0ffdee19a9fe"
Jul 17 00:55:39.684281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:39.684254    2573 scope.go:117] "RemoveContainer" containerID="0a6835f6b7aae7eb3301a38310d07ed55ab22f773876d526c11d14793019b81b"
Jul 17 00:55:39.684628 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:39.684429    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-kvl5l_odh-ai-gateway-infra(500540df-576a-4499-88b7-0ffdee19a9fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" podUID="500540df-576a-4499-88b7-0ffdee19a9fe"
Jul 17 00:55:46.962770 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:46.962744    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:55:46.963170 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:46.962925    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:55:49.962623 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:49.962591    2573 scope.go:117] "RemoveContainer" containerID="3c780237cd19b09977f83b55b0ec47412423e3d527783e4ac84f1d3803153b3f"
Jul 17 00:55:50.521903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.521868    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"]
Jul 17 00:55:50.525264 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.525246    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"
Jul 17 00:55:50.533207 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.533180    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"]
Jul 17 00:55:50.589983 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.589952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b98cb85-f352-4744-a861-0fd5c3ef87f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf\" (UID: \"2b98cb85-f352-4744-a861-0fd5c3ef87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"
Jul 17 00:55:50.690655 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.690635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b98cb85-f352-4744-a861-0fd5c3ef87f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf\" (UID: \"2b98cb85-f352-4744-a861-0fd5c3ef87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"
Jul 17 00:55:50.691254 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.691236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b98cb85-f352-4744-a861-0fd5c3ef87f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf\" (UID: \"2b98cb85-f352-4744-a861-0fd5c3ef87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"
Jul 17 00:55:50.743062 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.743036    2573 generic.go:356] "Generic (PLEG): container finished" podID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerID="8e61150f7e2c3505ad297ec0a43153c56709214c8bf53f57f5454f8abbdf10fc" exitCode=6
Jul 17 00:55:50.743174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.743099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" event={"ID":"cdfbb015-395d-43e5-99d8-dc9b2c30f677","Type":"ContainerDied","Data":"8e61150f7e2c3505ad297ec0a43153c56709214c8bf53f57f5454f8abbdf10fc"}
Jul 17 00:55:50.743174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.743160    2573 scope.go:117] "RemoveContainer" containerID="3c780237cd19b09977f83b55b0ec47412423e3d527783e4ac84f1d3803153b3f"
Jul 17 00:55:50.743442 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.743421    2573 scope.go:117] "RemoveContainer" containerID="8e61150f7e2c3505ad297ec0a43153c56709214c8bf53f57f5454f8abbdf10fc"
Jul 17 00:55:50.743645 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:50.743625    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s_odh-ai-gateway-infra(cdfbb015-395d-43e5-99d8-dc9b2c30f677)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677"
Jul 17 00:55:50.836247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.836196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"
Jul 17 00:55:50.962603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.962583    2573 scope.go:117] "RemoveContainer" containerID="0a6835f6b7aae7eb3301a38310d07ed55ab22f773876d526c11d14793019b81b"
Jul 17 00:55:50.962814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:50.962776    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"]
Jul 17 00:55:50.965267 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:55:50.965233    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b98cb85_f352_4744_a861_0fd5c3ef87f8.slice/crio-ce61ed1a4a2f87cb749798d52975e703d8041e7c48303706acdb5f0ab84a1ceb WatchSource:0}: Error finding container ce61ed1a4a2f87cb749798d52975e703d8041e7c48303706acdb5f0ab84a1ceb: Status 404 returned error can't find the container with id ce61ed1a4a2f87cb749798d52975e703d8041e7c48303706acdb5f0ab84a1ceb
Jul 17 00:55:51.751740 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.751711    2573 generic.go:356] "Generic (PLEG): container finished" podID="500540df-576a-4499-88b7-0ffdee19a9fe" containerID="d1e11c8e023c2235e3b500437086bfd4bf16424cae21e7e7a9f098da01a3bbe6" exitCode=7
Jul 17 00:55:51.751857 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.751778    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" event={"ID":"500540df-576a-4499-88b7-0ffdee19a9fe","Type":"ContainerDied","Data":"d1e11c8e023c2235e3b500437086bfd4bf16424cae21e7e7a9f098da01a3bbe6"}
Jul 17 00:55:51.751912 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.751881    2573 scope.go:117] "RemoveContainer" containerID="0a6835f6b7aae7eb3301a38310d07ed55ab22f773876d526c11d14793019b81b"
Jul 17 00:55:51.752160 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.752143    2573 scope.go:117] "RemoveContainer" containerID="d1e11c8e023c2235e3b500437086bfd4bf16424cae21e7e7a9f098da01a3bbe6"
Jul 17 00:55:51.752351 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:51.752330    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-kvl5l_odh-ai-gateway-infra(500540df-576a-4499-88b7-0ffdee19a9fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" podUID="500540df-576a-4499-88b7-0ffdee19a9fe"
Jul 17 00:55:51.753413 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.753382    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerID="b01d323110bd169c90b0619e96039fccec7d8f958a905d425db68c94531150fe" exitCode=6
Jul 17 00:55:51.753497 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.753411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" event={"ID":"2b98cb85-f352-4744-a861-0fd5c3ef87f8","Type":"ContainerDied","Data":"b01d323110bd169c90b0619e96039fccec7d8f958a905d425db68c94531150fe"}
Jul 17 00:55:51.753497 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.753471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" event={"ID":"2b98cb85-f352-4744-a861-0fd5c3ef87f8","Type":"ContainerStarted","Data":"ce61ed1a4a2f87cb749798d52975e703d8041e7c48303706acdb5f0ab84a1ceb"}
Jul 17 00:55:51.753626 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.753614    2573 scope.go:117] "RemoveContainer" containerID="b01d323110bd169c90b0619e96039fccec7d8f958a905d425db68c94531150fe"
Jul 17 00:55:51.766169 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.766135    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"]
Jul 17 00:55:51.889541 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.889519    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"
Jul 17 00:55:51.901163 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.901142    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfbb015-395d-43e5-99d8-dc9b2c30f677-maas-api-service-ca\") pod \"cdfbb015-395d-43e5-99d8-dc9b2c30f677\" (UID: \"cdfbb015-395d-43e5-99d8-dc9b2c30f677\") "
Jul 17 00:55:51.901455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:51.901435    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfbb015-395d-43e5-99d8-dc9b2c30f677-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cdfbb015-395d-43e5-99d8-dc9b2c30f677" (UID: "cdfbb015-395d-43e5-99d8-dc9b2c30f677"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:55:52.001858 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.001832    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfbb015-395d-43e5-99d8-dc9b2c30f677-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:55:52.760435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.760409    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s" event={"ID":"cdfbb015-395d-43e5-99d8-dc9b2c30f677","Type":"ContainerDied","Data":"d1f372ec7c42d1dfb348cb36cfebd08b53fab5ae46e5fe5a3f005dc8a4d8c9a7"}
Jul 17 00:55:52.760596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.760432    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"
Jul 17 00:55:52.760596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.760485    2573 scope.go:117] "RemoveContainer" containerID="8e61150f7e2c3505ad297ec0a43153c56709214c8bf53f57f5454f8abbdf10fc"
Jul 17 00:55:52.764711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.764681    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerID="16bed2d498c876ec31f214d1c979159e5136c660d0a4fc69f4bb3c4cceab41df" exitCode=6
Jul 17 00:55:52.764841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.764756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" event={"ID":"2b98cb85-f352-4744-a861-0fd5c3ef87f8","Type":"ContainerDied","Data":"16bed2d498c876ec31f214d1c979159e5136c660d0a4fc69f4bb3c4cceab41df"}
Jul 17 00:55:52.764982 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.764967    2573 scope.go:117] "RemoveContainer" containerID="16bed2d498c876ec31f214d1c979159e5136c660d0a4fc69f4bb3c4cceab41df"
Jul 17 00:55:52.765157 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:52.765139    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf_odh-ai-gateway-infra(2b98cb85-f352-4744-a861-0fd5c3ef87f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8"
Jul 17 00:55:52.769624 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.769598    2573 scope.go:117] "RemoveContainer" containerID="b01d323110bd169c90b0619e96039fccec7d8f958a905d425db68c94531150fe"
Jul 17 00:55:52.779083 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.779045    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"]
Jul 17 00:55:52.781370 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.781351    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"]
Jul 17 00:55:52.787095 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.787068    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-brp4s"]
Jul 17 00:55:52.912882 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:52.912861    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"
Jul 17 00:55:53.008007 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.007976    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500540df-576a-4499-88b7-0ffdee19a9fe-maas-api-service-ca\") pod \"500540df-576a-4499-88b7-0ffdee19a9fe\" (UID: \"500540df-576a-4499-88b7-0ffdee19a9fe\") "
Jul 17 00:55:53.008374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.008263    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500540df-576a-4499-88b7-0ffdee19a9fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "500540df-576a-4499-88b7-0ffdee19a9fe" (UID: "500540df-576a-4499-88b7-0ffdee19a9fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:55:53.008374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.008331    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500540df-576a-4499-88b7-0ffdee19a9fe-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:55:53.771351 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.771323    2573 scope.go:117] "RemoveContainer" containerID="16bed2d498c876ec31f214d1c979159e5136c660d0a4fc69f4bb3c4cceab41df"
Jul 17 00:55:53.771570 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:55:53.771548    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf_odh-ai-gateway-infra(2b98cb85-f352-4744-a861-0fd5c3ef87f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8"
Jul 17 00:55:53.773317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.773299    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"
Jul 17 00:55:53.773317 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.773306    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l" event={"ID":"500540df-576a-4499-88b7-0ffdee19a9fe","Type":"ContainerDied","Data":"a436d89b6d158c1ac4f1316f597d39aee7a899866c97d706df5a1f3306324bdd"}
Jul 17 00:55:53.773441 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.773366    2573 scope.go:117] "RemoveContainer" containerID="d1e11c8e023c2235e3b500437086bfd4bf16424cae21e7e7a9f098da01a3bbe6"
Jul 17 00:55:53.799821 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.799782    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"]
Jul 17 00:55:53.801967 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.801935    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-kvl5l"]
Jul 17 00:55:53.968580 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.968547    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" path="/var/lib/kubelet/pods/500540df-576a-4499-88b7-0ffdee19a9fe/volumes"
Jul 17 00:55:53.968917 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:53.968903    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" path="/var/lib/kubelet/pods/cdfbb015-395d-43e5-99d8-dc9b2c30f677/volumes"
Jul 17 00:55:58.699335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699298    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"]
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699640    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699652    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699670    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699676    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699683    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699689    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699698    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699703    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699709    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.699711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699714    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.700102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699724    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.700102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699729    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.700102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699805    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.700102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699811    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.700102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699817    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:55:58.700102 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.699824    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:55:58.704372 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.704355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"
Jul 17 00:55:58.708665 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.708635    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"]
Jul 17 00:55:58.750299 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.750277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c10566-5796-49d6-bb5d-2a4a5fa8ee8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-z95vk\" (UID: \"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"
Jul 17 00:55:58.851213 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.851187    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c10566-5796-49d6-bb5d-2a4a5fa8ee8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-z95vk\" (UID: \"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"
Jul 17 00:55:58.851765 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.851743    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c10566-5796-49d6-bb5d-2a4a5fa8ee8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-z95vk\" (UID: \"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"
Jul 17 00:55:58.962416 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:58.962358    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:55:59.016747 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:59.016723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"
Jul 17 00:55:59.352162 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:59.352137    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"]
Jul 17 00:55:59.353619 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:55:59.353591    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c10566_5796_49d6_bb5d_2a4a5fa8ee8a.slice/crio-cc3577c2f2f1c9965aa8b6208f677834256e932d9d8725c813a3fcd04628f278 WatchSource:0}: Error finding container cc3577c2f2f1c9965aa8b6208f677834256e932d9d8725c813a3fcd04628f278: Status 404 returned error can't find the container with id cc3577c2f2f1c9965aa8b6208f677834256e932d9d8725c813a3fcd04628f278
Jul 17 00:55:59.817230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:59.817196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerStarted","Data":"348ce1dece7a144e0243dbc0b52f40b5ec2ab08b8fc126af11ede931d4cde984"}
Jul 17 00:55:59.817687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:59.817477    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:55:59.818909 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:59.818886    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" event={"ID":"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a","Type":"ContainerStarted","Data":"3f81d340c01d25a2ea47a0564ea8ee435147a51175114f28897e883f53015854"}
Jul 17 00:55:59.819048 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:59.818937    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" event={"ID":"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a","Type":"ContainerStarted","Data":"cc3577c2f2f1c9965aa8b6208f677834256e932d9d8725c813a3fcd04628f278"}
Jul 17 00:55:59.849181 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:55:59.849137    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" podStartSLOduration=1.849122519 podStartE2EDuration="1.849122519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:55:59.847233714 +0000 UTC m=+2354.423386289" watchObservedRunningTime="2026-07-17 00:55:59.849122519 +0000 UTC m=+2354.425275096"
Jul 17 00:56:00.826805 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:00.826748    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerID="348ce1dece7a144e0243dbc0b52f40b5ec2ab08b8fc126af11ede931d4cde984" exitCode=1
Jul 17 00:56:00.826805 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:00.826803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerDied","Data":"348ce1dece7a144e0243dbc0b52f40b5ec2ab08b8fc126af11ede931d4cde984"}
Jul 17 00:56:00.827309 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:00.826862    2573 scope.go:117] "RemoveContainer" containerID="a3919bce01907b3fd5ec2c6431ba8731cad38fb2e95fe0ba8c38fc04825e1899"
Jul 17 00:56:00.827309 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:00.827129    2573 scope.go:117] "RemoveContainer" containerID="348ce1dece7a144e0243dbc0b52f40b5ec2ab08b8fc126af11ede931d4cde984"
Jul 17 00:56:00.827406 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:00.827364    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:56:00.828436 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:00.828412    2573 generic.go:356] "Generic (PLEG): container finished" podID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerID="3f81d340c01d25a2ea47a0564ea8ee435147a51175114f28897e883f53015854" exitCode=6
Jul 17 00:56:00.828543 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:00.828499    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" event={"ID":"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a","Type":"ContainerDied","Data":"3f81d340c01d25a2ea47a0564ea8ee435147a51175114f28897e883f53015854"}
Jul 17 00:56:00.828617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:00.828602    2573 scope.go:117] "RemoveContainer" containerID="3f81d340c01d25a2ea47a0564ea8ee435147a51175114f28897e883f53015854"
Jul 17 00:56:01.834984 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:01.834953    2573 scope.go:117] "RemoveContainer" containerID="348ce1dece7a144e0243dbc0b52f40b5ec2ab08b8fc126af11ede931d4cde984"
Jul 17 00:56:01.835435 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:01.835177    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:56:01.836530 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:01.836504    2573 generic.go:356] "Generic (PLEG): container finished" podID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerID="d3272672e49f716a10c9b91df23f437ed205d7f2a98cd23a5de58cb812ac11e8" exitCode=6
Jul 17 00:56:01.836670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:01.836561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" event={"ID":"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a","Type":"ContainerDied","Data":"d3272672e49f716a10c9b91df23f437ed205d7f2a98cd23a5de58cb812ac11e8"}
Jul 17 00:56:01.836670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:01.836615    2573 scope.go:117] "RemoveContainer" containerID="3f81d340c01d25a2ea47a0564ea8ee435147a51175114f28897e883f53015854"
Jul 17 00:56:01.836847 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:01.836829    2573 scope.go:117] "RemoveContainer" containerID="d3272672e49f716a10c9b91df23f437ed205d7f2a98cd23a5de58cb812ac11e8"
Jul 17 00:56:01.837016 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:01.836999    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-z95vk_odh-ai-gateway-infra(81c10566-5796-49d6-bb5d-2a4a5fa8ee8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a"
Jul 17 00:56:02.419532 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:02.419500    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:56:02.843802 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:02.843759    2573 scope.go:117] "RemoveContainer" containerID="d3272672e49f716a10c9b91df23f437ed205d7f2a98cd23a5de58cb812ac11e8"
Jul 17 00:56:02.844180 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:02.843900    2573 scope.go:117] "RemoveContainer" containerID="348ce1dece7a144e0243dbc0b52f40b5ec2ab08b8fc126af11ede931d4cde984"
Jul 17 00:56:02.844180 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:02.843980    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-z95vk_odh-ai-gateway-infra(81c10566-5796-49d6-bb5d-2a4a5fa8ee8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a"
Jul 17 00:56:02.844180 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:02.844028    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-732542a6-bb64878bf-pktwf_odh-ai-gateway-infra(7e3af469-52c3-4228-bc6f-ee97a64ace7e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e"
Jul 17 00:56:06.962690 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:06.962658    2573 scope.go:117] "RemoveContainer" containerID="16bed2d498c876ec31f214d1c979159e5136c660d0a4fc69f4bb3c4cceab41df"
Jul 17 00:56:07.872669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:07.872591    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerID="d9eecaf40f366a86f21de968ac4e789002d4ee2d9fd23008979a8a6303eff330" exitCode=6
Jul 17 00:56:07.872669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:07.872631    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" event={"ID":"2b98cb85-f352-4744-a861-0fd5c3ef87f8","Type":"ContainerDied","Data":"d9eecaf40f366a86f21de968ac4e789002d4ee2d9fd23008979a8a6303eff330"}
Jul 17 00:56:07.872903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:07.872679    2573 scope.go:117] "RemoveContainer" containerID="16bed2d498c876ec31f214d1c979159e5136c660d0a4fc69f4bb3c4cceab41df"
Jul 17 00:56:07.873009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:07.872985    2573 scope.go:117] "RemoveContainer" containerID="d9eecaf40f366a86f21de968ac4e789002d4ee2d9fd23008979a8a6303eff330"
Jul 17 00:56:07.873254 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:07.873228    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf_odh-ai-gateway-infra(2b98cb85-f352-4744-a861-0fd5c3ef87f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8"
Jul 17 00:56:08.895614 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:08.895579    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"]
Jul 17 00:56:09.028829 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.028785    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"
Jul 17 00:56:09.141056 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.141021    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b98cb85-f352-4744-a861-0fd5c3ef87f8-maas-api-service-ca\") pod \"2b98cb85-f352-4744-a861-0fd5c3ef87f8\" (UID: \"2b98cb85-f352-4744-a861-0fd5c3ef87f8\") "
Jul 17 00:56:09.141467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.141439    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b98cb85-f352-4744-a861-0fd5c3ef87f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b98cb85-f352-4744-a861-0fd5c3ef87f8" (UID: "2b98cb85-f352-4744-a861-0fd5c3ef87f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:56:09.241832 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.241735    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b98cb85-f352-4744-a861-0fd5c3ef87f8-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:09.885782 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.885753    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"
Jul 17 00:56:09.886009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.885818    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf" event={"ID":"2b98cb85-f352-4744-a861-0fd5c3ef87f8","Type":"ContainerDied","Data":"ce61ed1a4a2f87cb749798d52975e703d8041e7c48303706acdb5f0ab84a1ceb"}
Jul 17 00:56:09.886009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.885917    2573 scope.go:117] "RemoveContainer" containerID="d9eecaf40f366a86f21de968ac4e789002d4ee2d9fd23008979a8a6303eff330"
Jul 17 00:56:09.906485 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.906458    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"]
Jul 17 00:56:09.911881 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.911854    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-8xnqf"]
Jul 17 00:56:09.967559 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:09.967528    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" path="/var/lib/kubelet/pods/2b98cb85-f352-4744-a861-0fd5c3ef87f8/volumes"
Jul 17 00:56:11.761348 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.761312    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"]
Jul 17 00:56:11.899688 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.899653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf" event={"ID":"7e3af469-52c3-4228-bc6f-ee97a64ace7e","Type":"ContainerDied","Data":"83ea409f14560ddf32a310b088fcfd509651f5bfdcaaf593068e94f79b75df0f"}
Jul 17 00:56:11.899858 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.899707    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ea409f14560ddf32a310b088fcfd509651f5bfdcaaf593068e94f79b75df0f"
Jul 17 00:56:11.904658 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.904636    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:56:11.969196 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.969165    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsqtf\" (UniqueName: \"kubernetes.io/projected/7e3af469-52c3-4228-bc6f-ee97a64ace7e-kube-api-access-bsqtf\") pod \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") "
Jul 17 00:56:11.969376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.969219    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls\") pod \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") "
Jul 17 00:56:11.969376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.969274    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3af469-52c3-4228-bc6f-ee97a64ace7e-openshift-service-ca\") pod \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\" (UID: \"7e3af469-52c3-4228-bc6f-ee97a64ace7e\") "
Jul 17 00:56:11.969667 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.969639    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3af469-52c3-4228-bc6f-ee97a64ace7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e3af469-52c3-4228-bc6f-ee97a64ace7e" (UID: "7e3af469-52c3-4228-bc6f-ee97a64ace7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:56:11.971278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.971256    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e3af469-52c3-4228-bc6f-ee97a64ace7e" (UID: "7e3af469-52c3-4228-bc6f-ee97a64ace7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:56:11.971278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:11.971268    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3af469-52c3-4228-bc6f-ee97a64ace7e-kube-api-access-bsqtf" (OuterVolumeSpecName: "kube-api-access-bsqtf") pod "7e3af469-52c3-4228-bc6f-ee97a64ace7e" (UID: "7e3af469-52c3-4228-bc6f-ee97a64ace7e"). InnerVolumeSpecName "kube-api-access-bsqtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:56:12.070396 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:12.070304    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsqtf\" (UniqueName: \"kubernetes.io/projected/7e3af469-52c3-4228-bc6f-ee97a64ace7e-kube-api-access-bsqtf\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:12.070396 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:12.070335    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3af469-52c3-4228-bc6f-ee97a64ace7e-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:12.070396 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:12.070347    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3af469-52c3-4228-bc6f-ee97a64ace7e-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:12.904709 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:12.904675    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"
Jul 17 00:56:12.926268 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:12.926244    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"]
Jul 17 00:56:12.930383 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:12.930358    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-732542a6-bb64878bf-pktwf"]
Jul 17 00:56:13.967355 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:13.967314    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" path="/var/lib/kubelet/pods/7e3af469-52c3-4228-bc6f-ee97a64ace7e/volumes"
Jul 17 00:56:16.962993 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:16.962961    2573 scope.go:117] "RemoveContainer" containerID="d3272672e49f716a10c9b91df23f437ed205d7f2a98cd23a5de58cb812ac11e8"
Jul 17 00:56:17.932596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:17.932509    2573 generic.go:356] "Generic (PLEG): container finished" podID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerID="cadb50891198918bf48d528fc878fbb4f76a67a7ff338a83419ab1bf890cf49e" exitCode=6
Jul 17 00:56:17.932596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:17.932555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" event={"ID":"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a","Type":"ContainerDied","Data":"cadb50891198918bf48d528fc878fbb4f76a67a7ff338a83419ab1bf890cf49e"}
Jul 17 00:56:17.932857 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:17.932614    2573 scope.go:117] "RemoveContainer" containerID="d3272672e49f716a10c9b91df23f437ed205d7f2a98cd23a5de58cb812ac11e8"
Jul 17 00:56:17.933027 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:17.933006    2573 scope.go:117] "RemoveContainer" containerID="cadb50891198918bf48d528fc878fbb4f76a67a7ff338a83419ab1bf890cf49e"
Jul 17 00:56:17.933250 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:17.933224    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-z95vk_odh-ai-gateway-infra(81c10566-5796-49d6-bb5d-2a4a5fa8ee8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a"
Jul 17 00:56:18.957890 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:18.957847    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"]
Jul 17 00:56:19.094455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.094431    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"
Jul 17 00:56:19.238871 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.238804    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c10566-5796-49d6-bb5d-2a4a5fa8ee8a-maas-api-service-ca\") pod \"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a\" (UID: \"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a\") "
Jul 17 00:56:19.239222 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.239194    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c10566-5796-49d6-bb5d-2a4a5fa8ee8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" (UID: "81c10566-5796-49d6-bb5d-2a4a5fa8ee8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:56:19.339664 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.339639    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c10566-5796-49d6-bb5d-2a4a5fa8ee8a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:19.946716 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.946686    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"
Jul 17 00:56:19.946716 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.946701    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk" event={"ID":"81c10566-5796-49d6-bb5d-2a4a5fa8ee8a","Type":"ContainerDied","Data":"cc3577c2f2f1c9965aa8b6208f677834256e932d9d8725c813a3fcd04628f278"}
Jul 17 00:56:19.946996 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.946768    2573 scope.go:117] "RemoveContainer" containerID="cadb50891198918bf48d528fc878fbb4f76a67a7ff338a83419ab1bf890cf49e"
Jul 17 00:56:19.969604 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.969580    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"]
Jul 17 00:56:19.977105 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:19.977084    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z95vk"]
Jul 17 00:56:21.967341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:21.967304    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" path="/var/lib/kubelet/pods/81c10566-5796-49d6-bb5d-2a4a5fa8ee8a/volumes"
Jul 17 00:56:22.998931 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.998901    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"]
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999225    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999237    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999247    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999253    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999263    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999269    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999279    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999284    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999293    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999299    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999308    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999314    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999324    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999329    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999337    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999342    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999347    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999352    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999360    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999365    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999421    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfbb015-395d-43e5-99d8-dc9b2c30f677" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999428    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999434    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999439    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999445    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999449    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999454    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999459    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999465    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c10566-5796-49d6-bb5d-2a4a5fa8ee8a" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999470    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999476    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="500540df-576a-4499-88b7-0ffdee19a9fe" containerName="revoke-keys"
Jul 17 00:56:22.999645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:22.999481    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b98cb85-f352-4744-a861-0fd5c3ef87f8" containerName="revoke-keys"
Jul 17 00:56:23.002509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.002486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"
Jul 17 00:56:23.004519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.004498    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:56:23.009097 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.009075    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"]
Jul 17 00:56:23.069426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.069395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4546ddef-b42f-4733-8e33-9635b243feca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k\" (UID: \"4546ddef-b42f-4733-8e33-9635b243feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"
Jul 17 00:56:23.170226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.170202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4546ddef-b42f-4733-8e33-9635b243feca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k\" (UID: \"4546ddef-b42f-4733-8e33-9635b243feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"
Jul 17 00:56:23.170768 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.170748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4546ddef-b42f-4733-8e33-9635b243feca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k\" (UID: \"4546ddef-b42f-4733-8e33-9635b243feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"
Jul 17 00:56:23.314354 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.314332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"
Jul 17 00:56:23.457287 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.457200    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"]
Jul 17 00:56:23.458071 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:56:23.458042    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4546ddef_b42f_4733_8e33_9635b243feca.slice/crio-2ca53305c745ff2d38da286c68a9dedc2525cb9e80c2f7ec57fdb732c2779de3 WatchSource:0}: Error finding container 2ca53305c745ff2d38da286c68a9dedc2525cb9e80c2f7ec57fdb732c2779de3: Status 404 returned error can't find the container with id 2ca53305c745ff2d38da286c68a9dedc2525cb9e80c2f7ec57fdb732c2779de3
Jul 17 00:56:23.971925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.971833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" event={"ID":"4546ddef-b42f-4733-8e33-9635b243feca","Type":"ContainerStarted","Data":"bf477f49fb786fd652c15dcf7cc3c8d863b244787c282cdcb07ab3d66e4023ba"}
Jul 17 00:56:23.971925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.971887    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" event={"ID":"4546ddef-b42f-4733-8e33-9635b243feca","Type":"ContainerStarted","Data":"2ca53305c745ff2d38da286c68a9dedc2525cb9e80c2f7ec57fdb732c2779de3"}
Jul 17 00:56:23.986189 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:23.986140    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" podStartSLOduration=1.9861258130000001 podStartE2EDuration="1.986125813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:56:23.98391349 +0000 UTC m=+2378.560066061" watchObservedRunningTime="2026-07-17 00:56:23.986125813 +0000 UTC m=+2378.562278386"
Jul 17 00:56:24.978851 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:24.978810    2573 generic.go:356] "Generic (PLEG): container finished" podID="4546ddef-b42f-4733-8e33-9635b243feca" containerID="bf477f49fb786fd652c15dcf7cc3c8d863b244787c282cdcb07ab3d66e4023ba" exitCode=6
Jul 17 00:56:24.979229 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:24.978898    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" event={"ID":"4546ddef-b42f-4733-8e33-9635b243feca","Type":"ContainerDied","Data":"bf477f49fb786fd652c15dcf7cc3c8d863b244787c282cdcb07ab3d66e4023ba"}
Jul 17 00:56:24.979229 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:24.979078    2573 scope.go:117] "RemoveContainer" containerID="bf477f49fb786fd652c15dcf7cc3c8d863b244787c282cdcb07ab3d66e4023ba"
Jul 17 00:56:25.992907 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:25.992823    2573 generic.go:356] "Generic (PLEG): container finished" podID="4546ddef-b42f-4733-8e33-9635b243feca" containerID="c1df3f603602bd86d98b9d69e005e1f2d7e97ba780792fa05b88c9d977e24efc" exitCode=6
Jul 17 00:56:25.993268 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:25.992907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" event={"ID":"4546ddef-b42f-4733-8e33-9635b243feca","Type":"ContainerDied","Data":"c1df3f603602bd86d98b9d69e005e1f2d7e97ba780792fa05b88c9d977e24efc"}
Jul 17 00:56:25.993268 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:25.992988    2573 scope.go:117] "RemoveContainer" containerID="bf477f49fb786fd652c15dcf7cc3c8d863b244787c282cdcb07ab3d66e4023ba"
Jul 17 00:56:25.993268 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:25.993068    2573 scope.go:117] "RemoveContainer" containerID="c1df3f603602bd86d98b9d69e005e1f2d7e97ba780792fa05b88c9d977e24efc"
Jul 17 00:56:25.993268 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:25.993257    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k_odh-ai-gateway-infra(4546ddef-b42f-4733-8e33-9635b243feca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" podUID="4546ddef-b42f-4733-8e33-9635b243feca"
Jul 17 00:56:26.008704 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.008680    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"]
Jul 17 00:56:26.009097 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.009081    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:26.009097 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.009099    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:26.009279 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.009109    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:26.009279 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.009117    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:26.009279 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.009207    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:26.013670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.013651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"
Jul 17 00:56:26.018500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.018477    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"]
Jul 17 00:56:26.094689 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.094647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69647bc-1e5a-4650-9816-586d2f86f736-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-cvglq\" (UID: \"a69647bc-1e5a-4650-9816-586d2f86f736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"
Jul 17 00:56:26.196054 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.196022    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69647bc-1e5a-4650-9816-586d2f86f736-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-cvglq\" (UID: \"a69647bc-1e5a-4650-9816-586d2f86f736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"
Jul 17 00:56:26.196566 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.196547    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69647bc-1e5a-4650-9816-586d2f86f736-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-cvglq\" (UID: \"a69647bc-1e5a-4650-9816-586d2f86f736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"
Jul 17 00:56:26.325727 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.325699    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"
Jul 17 00:56:26.457467 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:26.457439    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"]
Jul 17 00:56:26.458847 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:56:26.458817    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69647bc_1e5a_4650_9816_586d2f86f736.slice/crio-a34339ef1910afbc22fabf4b69c3fdfaa41844025be3dd1ccc5eaf9c65e9b708 WatchSource:0}: Error finding container a34339ef1910afbc22fabf4b69c3fdfaa41844025be3dd1ccc5eaf9c65e9b708: Status 404 returned error can't find the container with id a34339ef1910afbc22fabf4b69c3fdfaa41844025be3dd1ccc5eaf9c65e9b708
Jul 17 00:56:27.000468 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:27.000365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" event={"ID":"a69647bc-1e5a-4650-9816-586d2f86f736","Type":"ContainerStarted","Data":"43d0a696cc28d275936a5c58510148a9e99ef16a898bef073635332371c2acc9"}
Jul 17 00:56:27.000468 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:27.000432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" event={"ID":"a69647bc-1e5a-4650-9816-586d2f86f736","Type":"ContainerStarted","Data":"a34339ef1910afbc22fabf4b69c3fdfaa41844025be3dd1ccc5eaf9c65e9b708"}
Jul 17 00:56:27.002176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:27.002157    2573 scope.go:117] "RemoveContainer" containerID="c1df3f603602bd86d98b9d69e005e1f2d7e97ba780792fa05b88c9d977e24efc"
Jul 17 00:56:27.002340 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:27.002322    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k_odh-ai-gateway-infra(4546ddef-b42f-4733-8e33-9635b243feca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" podUID="4546ddef-b42f-4733-8e33-9635b243feca"
Jul 17 00:56:27.014830 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:27.014758    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" podStartSLOduration=2.014740214 podStartE2EDuration="2.014740214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:56:27.014352593 +0000 UTC m=+2381.590505165" watchObservedRunningTime="2026-07-17 00:56:27.014740214 +0000 UTC m=+2381.590892788"
Jul 17 00:56:28.008075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:28.008036    2573 generic.go:356] "Generic (PLEG): container finished" podID="a69647bc-1e5a-4650-9816-586d2f86f736" containerID="43d0a696cc28d275936a5c58510148a9e99ef16a898bef073635332371c2acc9" exitCode=6
Jul 17 00:56:28.008075 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:28.008076    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" event={"ID":"a69647bc-1e5a-4650-9816-586d2f86f736","Type":"ContainerDied","Data":"43d0a696cc28d275936a5c58510148a9e99ef16a898bef073635332371c2acc9"}
Jul 17 00:56:28.008503 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:28.008306    2573 scope.go:117] "RemoveContainer" containerID="43d0a696cc28d275936a5c58510148a9e99ef16a898bef073635332371c2acc9"
Jul 17 00:56:29.015839 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:29.015810    2573 generic.go:356] "Generic (PLEG): container finished" podID="a69647bc-1e5a-4650-9816-586d2f86f736" containerID="e4a511455a82b44d2e10155b305af06f13e4db775caa46e16282091d39092d35" exitCode=6
Jul 17 00:56:29.016240 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:29.015888    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" event={"ID":"a69647bc-1e5a-4650-9816-586d2f86f736","Type":"ContainerDied","Data":"e4a511455a82b44d2e10155b305af06f13e4db775caa46e16282091d39092d35"}
Jul 17 00:56:29.016240 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:29.015939    2573 scope.go:117] "RemoveContainer" containerID="43d0a696cc28d275936a5c58510148a9e99ef16a898bef073635332371c2acc9"
Jul 17 00:56:29.016240 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:29.016160    2573 scope.go:117] "RemoveContainer" containerID="e4a511455a82b44d2e10155b305af06f13e4db775caa46e16282091d39092d35"
Jul 17 00:56:29.016398 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:29.016362    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-cvglq_odh-ai-gateway-infra(a69647bc-1e5a-4650-9816-586d2f86f736)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" podUID="a69647bc-1e5a-4650-9816-586d2f86f736"
Jul 17 00:56:30.023061 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:30.023033    2573 scope.go:117] "RemoveContainer" containerID="e4a511455a82b44d2e10155b305af06f13e4db775caa46e16282091d39092d35"
Jul 17 00:56:30.023533 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:30.023281    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-cvglq_odh-ai-gateway-infra(a69647bc-1e5a-4650-9816-586d2f86f736)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" podUID="a69647bc-1e5a-4650-9816-586d2f86f736"
Jul 17 00:56:39.968778 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:39.968746    2573 scope.go:117] "RemoveContainer" containerID="c1df3f603602bd86d98b9d69e005e1f2d7e97ba780792fa05b88c9d977e24efc"
Jul 17 00:56:40.560521 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.560489    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"]
Jul 17 00:56:40.560945 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.560932    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3af469-52c3-4228-bc6f-ee97a64ace7e" containerName="maas-api"
Jul 17 00:56:40.563845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.563825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"
Jul 17 00:56:40.569945 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.569914    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"]
Jul 17 00:56:40.623009 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.622980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045e8249-7027-4d07-bade-72c2150c4156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp\" (UID: \"045e8249-7027-4d07-bade-72c2150c4156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"
Jul 17 00:56:40.724104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.724075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045e8249-7027-4d07-bade-72c2150c4156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp\" (UID: \"045e8249-7027-4d07-bade-72c2150c4156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"
Jul 17 00:56:40.724638 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.724619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045e8249-7027-4d07-bade-72c2150c4156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp\" (UID: \"045e8249-7027-4d07-bade-72c2150c4156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"
Jul 17 00:56:40.912198 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.912096    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"
Jul 17 00:56:40.963422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:40.963392    2573 scope.go:117] "RemoveContainer" containerID="e4a511455a82b44d2e10155b305af06f13e4db775caa46e16282091d39092d35"
Jul 17 00:56:41.046215 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:41.046185    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"]
Jul 17 00:56:41.046959 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:56:41.046929    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045e8249_7027_4d07_bade_72c2150c4156.slice/crio-4b6041ac14ef6ad5eca242ddb9d93945ed602dec21fdb957d78df2455fa09571 WatchSource:0}: Error finding container 4b6041ac14ef6ad5eca242ddb9d93945ed602dec21fdb957d78df2455fa09571: Status 404 returned error can't find the container with id 4b6041ac14ef6ad5eca242ddb9d93945ed602dec21fdb957d78df2455fa09571
Jul 17 00:56:41.090813 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:41.090764    2573 generic.go:356] "Generic (PLEG): container finished" podID="4546ddef-b42f-4733-8e33-9635b243feca" containerID="cac47a98fcfc9eedd0361db7ee5bf384fd57f9e82aa4e6283ab7bdfb999c31e3" exitCode=6
Jul 17 00:56:41.090970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:41.090819    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" event={"ID":"4546ddef-b42f-4733-8e33-9635b243feca","Type":"ContainerDied","Data":"cac47a98fcfc9eedd0361db7ee5bf384fd57f9e82aa4e6283ab7bdfb999c31e3"}
Jul 17 00:56:41.090970 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:41.090872    2573 scope.go:117] "RemoveContainer" containerID="c1df3f603602bd86d98b9d69e005e1f2d7e97ba780792fa05b88c9d977e24efc"
Jul 17 00:56:41.091183 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:41.091165    2573 scope.go:117] "RemoveContainer" containerID="cac47a98fcfc9eedd0361db7ee5bf384fd57f9e82aa4e6283ab7bdfb999c31e3"
Jul 17 00:56:41.091433 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:41.091408    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k_odh-ai-gateway-infra(4546ddef-b42f-4733-8e33-9635b243feca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" podUID="4546ddef-b42f-4733-8e33-9635b243feca"
Jul 17 00:56:41.092156 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:41.092137    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" event={"ID":"045e8249-7027-4d07-bade-72c2150c4156","Type":"ContainerStarted","Data":"4b6041ac14ef6ad5eca242ddb9d93945ed602dec21fdb957d78df2455fa09571"}
Jul 17 00:56:42.098869 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.098835    2573 generic.go:356] "Generic (PLEG): container finished" podID="a69647bc-1e5a-4650-9816-586d2f86f736" containerID="fc903113cafb8fafcf626b5de8fe2ab31b3434fefc1dcb0a97a212737624f2fb" exitCode=6
Jul 17 00:56:42.099453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.098908    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" event={"ID":"a69647bc-1e5a-4650-9816-586d2f86f736","Type":"ContainerDied","Data":"fc903113cafb8fafcf626b5de8fe2ab31b3434fefc1dcb0a97a212737624f2fb"}
Jul 17 00:56:42.099453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.098969    2573 scope.go:117] "RemoveContainer" containerID="e4a511455a82b44d2e10155b305af06f13e4db775caa46e16282091d39092d35"
Jul 17 00:56:42.099453 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.099295    2573 scope.go:117] "RemoveContainer" containerID="fc903113cafb8fafcf626b5de8fe2ab31b3434fefc1dcb0a97a212737624f2fb"
Jul 17 00:56:42.099609 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:42.099523    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-cvglq_odh-ai-gateway-infra(a69647bc-1e5a-4650-9816-586d2f86f736)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" podUID="a69647bc-1e5a-4650-9816-586d2f86f736"
Jul 17 00:56:42.101835 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.101806    2573 generic.go:356] "Generic (PLEG): container finished" podID="045e8249-7027-4d07-bade-72c2150c4156" containerID="edd6363a530d3be28de6f412039cd627172d9f6d9f000977f20812b8eeda0526" exitCode=6
Jul 17 00:56:42.101946 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.101845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" event={"ID":"045e8249-7027-4d07-bade-72c2150c4156","Type":"ContainerDied","Data":"edd6363a530d3be28de6f412039cd627172d9f6d9f000977f20812b8eeda0526"}
Jul 17 00:56:42.102034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.102022    2573 scope.go:117] "RemoveContainer" containerID="edd6363a530d3be28de6f412039cd627172d9f6d9f000977f20812b8eeda0526"
Jul 17 00:56:42.117325 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.117298    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"]
Jul 17 00:56:42.262576 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.262555    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"
Jul 17 00:56:42.337656 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.337625    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4546ddef-b42f-4733-8e33-9635b243feca-maas-api-service-ca\") pod \"4546ddef-b42f-4733-8e33-9635b243feca\" (UID: \"4546ddef-b42f-4733-8e33-9635b243feca\") "
Jul 17 00:56:42.338019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.337997    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4546ddef-b42f-4733-8e33-9635b243feca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4546ddef-b42f-4733-8e33-9635b243feca" (UID: "4546ddef-b42f-4733-8e33-9635b243feca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:56:42.438914 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:42.438878    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4546ddef-b42f-4733-8e33-9635b243feca-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:43.112391 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.112359    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"
Jul 17 00:56:43.112871 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.112387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k" event={"ID":"4546ddef-b42f-4733-8e33-9635b243feca","Type":"ContainerDied","Data":"2ca53305c745ff2d38da286c68a9dedc2525cb9e80c2f7ec57fdb732c2779de3"}
Jul 17 00:56:43.112871 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.112453    2573 scope.go:117] "RemoveContainer" containerID="cac47a98fcfc9eedd0361db7ee5bf384fd57f9e82aa4e6283ab7bdfb999c31e3"
Jul 17 00:56:43.114609 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.114580    2573 generic.go:356] "Generic (PLEG): container finished" podID="045e8249-7027-4d07-bade-72c2150c4156" containerID="3b82a5b5afa5d8a9673bf500364fc882ff8563caec288f6bf0eaea248f4c0210" exitCode=6
Jul 17 00:56:43.114748 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.114634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" event={"ID":"045e8249-7027-4d07-bade-72c2150c4156","Type":"ContainerDied","Data":"3b82a5b5afa5d8a9673bf500364fc882ff8563caec288f6bf0eaea248f4c0210"}
Jul 17 00:56:43.114884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.114868    2573 scope.go:117] "RemoveContainer" containerID="3b82a5b5afa5d8a9673bf500364fc882ff8563caec288f6bf0eaea248f4c0210"
Jul 17 00:56:43.115095 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:43.115072    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp_odh-ai-gateway-infra(045e8249-7027-4d07-bade-72c2150c4156)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" podUID="045e8249-7027-4d07-bade-72c2150c4156"
Jul 17 00:56:43.124354 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.124331    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"]
Jul 17 00:56:43.124483 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.124341    2573 scope.go:117] "RemoveContainer" containerID="edd6363a530d3be28de6f412039cd627172d9f6d9f000977f20812b8eeda0526"
Jul 17 00:56:43.143455 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.143430    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"]
Jul 17 00:56:43.148606 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.148583    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-mcj9k"]
Jul 17 00:56:43.265064 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.265043    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"
Jul 17 00:56:43.347490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.347454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69647bc-1e5a-4650-9816-586d2f86f736-maas-api-service-ca\") pod \"a69647bc-1e5a-4650-9816-586d2f86f736\" (UID: \"a69647bc-1e5a-4650-9816-586d2f86f736\") "
Jul 17 00:56:43.347841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.347819    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69647bc-1e5a-4650-9816-586d2f86f736-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a69647bc-1e5a-4650-9816-586d2f86f736" (UID: "a69647bc-1e5a-4650-9816-586d2f86f736"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:56:43.448511 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.448429    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69647bc-1e5a-4650-9816-586d2f86f736-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:43.968480 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:43.968446    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4546ddef-b42f-4733-8e33-9635b243feca" path="/var/lib/kubelet/pods/4546ddef-b42f-4733-8e33-9635b243feca/volumes"
Jul 17 00:56:44.121494 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:44.121469    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"
Jul 17 00:56:44.121863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:44.121488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq" event={"ID":"a69647bc-1e5a-4650-9816-586d2f86f736","Type":"ContainerDied","Data":"a34339ef1910afbc22fabf4b69c3fdfaa41844025be3dd1ccc5eaf9c65e9b708"}
Jul 17 00:56:44.121863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:44.121545    2573 scope.go:117] "RemoveContainer" containerID="fc903113cafb8fafcf626b5de8fe2ab31b3434fefc1dcb0a97a212737624f2fb"
Jul 17 00:56:44.124943 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:44.124922    2573 scope.go:117] "RemoveContainer" containerID="3b82a5b5afa5d8a9673bf500364fc882ff8563caec288f6bf0eaea248f4c0210"
Jul 17 00:56:44.125159 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:44.125134    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp_odh-ai-gateway-infra(045e8249-7027-4d07-bade-72c2150c4156)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" podUID="045e8249-7027-4d07-bade-72c2150c4156"
Jul 17 00:56:44.137222 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:44.137199    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"]
Jul 17 00:56:44.145692 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:44.145656    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-cvglq"]
Jul 17 00:56:45.969345 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:45.969318    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" path="/var/lib/kubelet/pods/a69647bc-1e5a-4650-9816-586d2f86f736/volumes"
Jul 17 00:56:47.871583 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:47.871462    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:56:47.891806 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:47.880620    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 00:56:54.962966 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:54.962936    2573 scope.go:117] "RemoveContainer" containerID="3b82a5b5afa5d8a9673bf500364fc882ff8563caec288f6bf0eaea248f4c0210"
Jul 17 00:56:56.190596 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:56.190562    2573 generic.go:356] "Generic (PLEG): container finished" podID="045e8249-7027-4d07-bade-72c2150c4156" containerID="7edf940dbeb7a2642eb7be969ffb23b95f7da4414b041161efef800e77f2c3f9" exitCode=6
Jul 17 00:56:56.191064 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:56.190628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" event={"ID":"045e8249-7027-4d07-bade-72c2150c4156","Type":"ContainerDied","Data":"7edf940dbeb7a2642eb7be969ffb23b95f7da4414b041161efef800e77f2c3f9"}
Jul 17 00:56:56.191064 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:56.190684    2573 scope.go:117] "RemoveContainer" containerID="3b82a5b5afa5d8a9673bf500364fc882ff8563caec288f6bf0eaea248f4c0210"
Jul 17 00:56:56.191064 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:56.190988    2573 scope.go:117] "RemoveContainer" containerID="7edf940dbeb7a2642eb7be969ffb23b95f7da4414b041161efef800e77f2c3f9"
Jul 17 00:56:56.191234 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:56:56.191202    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp_odh-ai-gateway-infra(045e8249-7027-4d07-bade-72c2150c4156)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" podUID="045e8249-7027-4d07-bade-72c2150c4156"
Jul 17 00:56:57.213670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:57.213639    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"]
Jul 17 00:56:57.354551 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:57.354528    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"
Jul 17 00:56:57.450229 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:57.450186    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045e8249-7027-4d07-bade-72c2150c4156-maas-api-service-ca\") pod \"045e8249-7027-4d07-bade-72c2150c4156\" (UID: \"045e8249-7027-4d07-bade-72c2150c4156\") "
Jul 17 00:56:57.450498 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:57.450477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045e8249-7027-4d07-bade-72c2150c4156-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "045e8249-7027-4d07-bade-72c2150c4156" (UID: "045e8249-7027-4d07-bade-72c2150c4156"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:56:57.550708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:57.550683    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045e8249-7027-4d07-bade-72c2150c4156-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:56:58.203668 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.203641    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"
Jul 17 00:56:58.203862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.203672    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp" event={"ID":"045e8249-7027-4d07-bade-72c2150c4156","Type":"ContainerDied","Data":"4b6041ac14ef6ad5eca242ddb9d93945ed602dec21fdb957d78df2455fa09571"}
Jul 17 00:56:58.203862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.203723    2573 scope.go:117] "RemoveContainer" containerID="7edf940dbeb7a2642eb7be969ffb23b95f7da4414b041161efef800e77f2c3f9"
Jul 17 00:56:58.218360 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.218338    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"]
Jul 17 00:56:58.232057 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.232006    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-bvswp"]
Jul 17 00:56:58.742802 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.742767    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"]
Jul 17 00:56:58.743173 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743158    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:56:58.743233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743175    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:56:58.743233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743183    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:56:58.743233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743199    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:56:58.743233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743213    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:56:58.743233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743220    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:56:58.743233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743229    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:56:58.743233 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743234    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743248    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743253    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743268    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743273    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743339    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743347    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743354    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743364    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743373    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:56:58.743469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.743380    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:56:58.747605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.747587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"
Jul 17 00:56:58.749769 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.749749    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:56:58.752148 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.752119    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"]
Jul 17 00:56:58.859041 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.859020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa228eb-3fd3-4810-85a4-5a31f5556d13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-k86wk\" (UID: \"5fa228eb-3fd3-4810-85a4-5a31f5556d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"
Jul 17 00:56:58.959780 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.959759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa228eb-3fd3-4810-85a4-5a31f5556d13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-k86wk\" (UID: \"5fa228eb-3fd3-4810-85a4-5a31f5556d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"
Jul 17 00:56:58.960337 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:58.960317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa228eb-3fd3-4810-85a4-5a31f5556d13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-k86wk\" (UID: \"5fa228eb-3fd3-4810-85a4-5a31f5556d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"
Jul 17 00:56:59.059776 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:59.059748    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"
Jul 17 00:56:59.192342 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:59.192318    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"]
Jul 17 00:56:59.194339 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:56:59.194305    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa228eb_3fd3_4810_85a4_5a31f5556d13.slice/crio-a223dd43eebf9827f173f2a93b696f3d658193d582cdc4f60aabff907e277106 WatchSource:0}: Error finding container a223dd43eebf9827f173f2a93b696f3d658193d582cdc4f60aabff907e277106: Status 404 returned error can't find the container with id a223dd43eebf9827f173f2a93b696f3d658193d582cdc4f60aabff907e277106
Jul 17 00:56:59.210461 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:59.210433    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" event={"ID":"5fa228eb-3fd3-4810-85a4-5a31f5556d13","Type":"ContainerStarted","Data":"a223dd43eebf9827f173f2a93b696f3d658193d582cdc4f60aabff907e277106"}
Jul 17 00:56:59.968529 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:56:59.968496    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="045e8249-7027-4d07-bade-72c2150c4156" path="/var/lib/kubelet/pods/045e8249-7027-4d07-bade-72c2150c4156/volumes"
Jul 17 00:57:00.218186 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:00.218156    2573 generic.go:356] "Generic (PLEG): container finished" podID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerID="e5bc6b3c5e36a4a4dc42dbb336ff064a1aeafda1aabe9d5caf69a3925c42eac6" exitCode=6
Jul 17 00:57:00.218371 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:00.218198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" event={"ID":"5fa228eb-3fd3-4810-85a4-5a31f5556d13","Type":"ContainerDied","Data":"e5bc6b3c5e36a4a4dc42dbb336ff064a1aeafda1aabe9d5caf69a3925c42eac6"}
Jul 17 00:57:00.218437 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:00.218424    2573 scope.go:117] "RemoveContainer" containerID="e5bc6b3c5e36a4a4dc42dbb336ff064a1aeafda1aabe9d5caf69a3925c42eac6"
Jul 17 00:57:01.225497 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:01.225409    2573 generic.go:356] "Generic (PLEG): container finished" podID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerID="fdc93bee1c49bcf0c3aff6d2f858f22409bdfe185b497fa50bc05fea7a83a635" exitCode=6
Jul 17 00:57:01.225893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:01.225496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" event={"ID":"5fa228eb-3fd3-4810-85a4-5a31f5556d13","Type":"ContainerDied","Data":"fdc93bee1c49bcf0c3aff6d2f858f22409bdfe185b497fa50bc05fea7a83a635"}
Jul 17 00:57:01.225893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:01.225550    2573 scope.go:117] "RemoveContainer" containerID="e5bc6b3c5e36a4a4dc42dbb336ff064a1aeafda1aabe9d5caf69a3925c42eac6"
Jul 17 00:57:01.225893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:01.225720    2573 scope.go:117] "RemoveContainer" containerID="fdc93bee1c49bcf0c3aff6d2f858f22409bdfe185b497fa50bc05fea7a83a635"
Jul 17 00:57:01.226095 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:01.225954    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-k86wk_odh-ai-gateway-infra(5fa228eb-3fd3-4810-85a4-5a31f5556d13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13"
Jul 17 00:57:02.231992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:02.231962    2573 scope.go:117] "RemoveContainer" containerID="fdc93bee1c49bcf0c3aff6d2f858f22409bdfe185b497fa50bc05fea7a83a635"
Jul 17 00:57:02.232353 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:02.232151    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-k86wk_odh-ai-gateway-infra(5fa228eb-3fd3-4810-85a4-5a31f5556d13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13"
Jul 17 00:57:12.962499 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:12.962407    2573 scope.go:117] "RemoveContainer" containerID="fdc93bee1c49bcf0c3aff6d2f858f22409bdfe185b497fa50bc05fea7a83a635"
Jul 17 00:57:13.293570 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:13.293538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" event={"ID":"5fa228eb-3fd3-4810-85a4-5a31f5556d13","Type":"ContainerStarted","Data":"f83cb8ff2d23a18284218872f1dc287ef84b0b31a6468ab4390f192abeac8a87"}
Jul 17 00:57:13.308013 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:13.307968    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" podStartSLOduration=15.30795296 podStartE2EDuration="15.30795296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:57:13.305812552 +0000 UTC m=+2427.881965128" watchObservedRunningTime="2026-07-17 00:57:13.30795296 +0000 UTC m=+2427.884105534"
Jul 17 00:57:14.301107 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:14.301068    2573 generic.go:356] "Generic (PLEG): container finished" podID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerID="f83cb8ff2d23a18284218872f1dc287ef84b0b31a6468ab4390f192abeac8a87" exitCode=6
Jul 17 00:57:14.301533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:14.301136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" event={"ID":"5fa228eb-3fd3-4810-85a4-5a31f5556d13","Type":"ContainerDied","Data":"f83cb8ff2d23a18284218872f1dc287ef84b0b31a6468ab4390f192abeac8a87"}
Jul 17 00:57:14.301533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:14.301194    2573 scope.go:117] "RemoveContainer" containerID="fdc93bee1c49bcf0c3aff6d2f858f22409bdfe185b497fa50bc05fea7a83a635"
Jul 17 00:57:14.301533 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:14.301430    2573 scope.go:117] "RemoveContainer" containerID="f83cb8ff2d23a18284218872f1dc287ef84b0b31a6468ab4390f192abeac8a87"
Jul 17 00:57:14.301674 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:14.301601    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-k86wk_odh-ai-gateway-infra(5fa228eb-3fd3-4810-85a4-5a31f5556d13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13"
Jul 17 00:57:14.317036 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:14.317015    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"]
Jul 17 00:57:15.437341 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:15.437316    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"
Jul 17 00:57:15.487980 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:15.487952    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa228eb-3fd3-4810-85a4-5a31f5556d13-maas-api-service-ca\") pod \"5fa228eb-3fd3-4810-85a4-5a31f5556d13\" (UID: \"5fa228eb-3fd3-4810-85a4-5a31f5556d13\") "
Jul 17 00:57:15.488265 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:15.488242    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa228eb-3fd3-4810-85a4-5a31f5556d13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fa228eb-3fd3-4810-85a4-5a31f5556d13" (UID: "5fa228eb-3fd3-4810-85a4-5a31f5556d13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:57:15.588906 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:15.588852    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa228eb-3fd3-4810-85a4-5a31f5556d13-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:57:16.043936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.043904    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"]
Jul 17 00:57:16.044249 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044236    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:57:16.044296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044250    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:57:16.044296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044263    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:16.044296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044268    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:16.044296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044278    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:57:16.044296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044283    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:57:16.044296 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044294    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044299    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044307    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044312    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044375    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="045e8249-7027-4d07-bade-72c2150c4156" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044382    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044388    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4546ddef-b42f-4733-8e33-9635b243feca" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044395    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a69647bc-1e5a-4650-9816-586d2f86f736" containerName="revoke-keys"
Jul 17 00:57:16.044479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.044400    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:16.047308 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.047290    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"
Jul 17 00:57:16.054122 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.054099    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"]
Jul 17 00:57:16.092827 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.092780    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d741221-1ce0-463f-a4e5-bfa1f0cb29ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-j7k7h\" (UID: \"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"
Jul 17 00:57:16.193855 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.193832    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d741221-1ce0-463f-a4e5-bfa1f0cb29ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-j7k7h\" (UID: \"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"
Jul 17 00:57:16.194396 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.194378    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d741221-1ce0-463f-a4e5-bfa1f0cb29ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-j7k7h\" (UID: \"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"
Jul 17 00:57:16.314519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.314496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk" event={"ID":"5fa228eb-3fd3-4810-85a4-5a31f5556d13","Type":"ContainerDied","Data":"a223dd43eebf9827f173f2a93b696f3d658193d582cdc4f60aabff907e277106"}
Jul 17 00:57:16.314666 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.314525    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"
Jul 17 00:57:16.314666 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.314551    2573 scope.go:117] "RemoveContainer" containerID="f83cb8ff2d23a18284218872f1dc287ef84b0b31a6468ab4390f192abeac8a87"
Jul 17 00:57:16.329090 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.329065    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"]
Jul 17 00:57:16.332565 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.332541    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-k86wk"]
Jul 17 00:57:16.358474 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.358455    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"
Jul 17 00:57:16.691876 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:16.691854    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"]
Jul 17 00:57:16.693748 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:57:16.693722    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d741221_1ce0_463f_a4e5_bfa1f0cb29ad.slice/crio-075bec4d61391f9730f3d51cff118148b5383a937bc5b8161af6db5f1d2bbaf3 WatchSource:0}: Error finding container 075bec4d61391f9730f3d51cff118148b5383a937bc5b8161af6db5f1d2bbaf3: Status 404 returned error can't find the container with id 075bec4d61391f9730f3d51cff118148b5383a937bc5b8161af6db5f1d2bbaf3
Jul 17 00:57:17.321510 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:17.321481    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerID="4261065c12fcc8eeae6fdc788b0df22beb7042872f3f94d36edb07a55713fbec" exitCode=6
Jul 17 00:57:17.321642 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:17.321570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" event={"ID":"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad","Type":"ContainerDied","Data":"4261065c12fcc8eeae6fdc788b0df22beb7042872f3f94d36edb07a55713fbec"}
Jul 17 00:57:17.321705 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:17.321674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" event={"ID":"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad","Type":"ContainerStarted","Data":"075bec4d61391f9730f3d51cff118148b5383a937bc5b8161af6db5f1d2bbaf3"}
Jul 17 00:57:17.321761 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:17.321714    2573 scope.go:117] "RemoveContainer" containerID="4261065c12fcc8eeae6fdc788b0df22beb7042872f3f94d36edb07a55713fbec"
Jul 17 00:57:17.968824 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:17.968782    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" path="/var/lib/kubelet/pods/5fa228eb-3fd3-4810-85a4-5a31f5556d13/volumes"
Jul 17 00:57:18.330479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:18.330451    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerID="b28140f79b974ddce935a783545dadddd45b946a8b0c4b7105d41d17aebbe192" exitCode=6
Jul 17 00:57:18.330636 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:18.330530    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" event={"ID":"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad","Type":"ContainerDied","Data":"b28140f79b974ddce935a783545dadddd45b946a8b0c4b7105d41d17aebbe192"}
Jul 17 00:57:18.330636 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:18.330588    2573 scope.go:117] "RemoveContainer" containerID="4261065c12fcc8eeae6fdc788b0df22beb7042872f3f94d36edb07a55713fbec"
Jul 17 00:57:18.330827 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:18.330782    2573 scope.go:117] "RemoveContainer" containerID="b28140f79b974ddce935a783545dadddd45b946a8b0c4b7105d41d17aebbe192"
Jul 17 00:57:18.331019 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:18.330997    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-j7k7h_odh-ai-gateway-infra(3d741221-1ce0-463f-a4e5-bfa1f0cb29ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad"
Jul 17 00:57:19.337711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:19.337685    2573 scope.go:117] "RemoveContainer" containerID="b28140f79b974ddce935a783545dadddd45b946a8b0c4b7105d41d17aebbe192"
Jul 17 00:57:19.338117 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:19.337918    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-j7k7h_odh-ai-gateway-infra(3d741221-1ce0-463f-a4e5-bfa1f0cb29ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad"
Jul 17 00:57:23.040834 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.040800    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"]
Jul 17 00:57:23.041200 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.041159    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:23.041200 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.041170    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:23.041276 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.041229    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa228eb-3fd3-4810-85a4-5a31f5556d13" containerName="revoke-keys"
Jul 17 00:57:23.044087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.044070    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"
Jul 17 00:57:23.049836 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.049811    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"]
Jul 17 00:57:23.146963 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.146941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dacf728-3d0a-45cb-bbc0-bf85419d744b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn\" (UID: \"1dacf728-3d0a-45cb-bbc0-bf85419d744b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"
Jul 17 00:57:23.247600 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.247577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dacf728-3d0a-45cb-bbc0-bf85419d744b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn\" (UID: \"1dacf728-3d0a-45cb-bbc0-bf85419d744b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"
Jul 17 00:57:23.248168 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.248151    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dacf728-3d0a-45cb-bbc0-bf85419d744b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn\" (UID: \"1dacf728-3d0a-45cb-bbc0-bf85419d744b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"
Jul 17 00:57:23.355925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.355873    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"
Jul 17 00:57:23.694375 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:23.694349    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"]
Jul 17 00:57:23.696873 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:57:23.696837    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dacf728_3d0a_45cb_bbc0_bf85419d744b.slice/crio-ffafc27926304105d5c7574a4b7f5bfe621c9d55db879a7508414035b496c7a2 WatchSource:0}: Error finding container ffafc27926304105d5c7574a4b7f5bfe621c9d55db879a7508414035b496c7a2: Status 404 returned error can't find the container with id ffafc27926304105d5c7574a4b7f5bfe621c9d55db879a7508414035b496c7a2
Jul 17 00:57:24.367195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:24.367165    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerID="2a6913201fad6e9b6df177419ab5b72d32b9796dac396813f879cff79feb92d1" exitCode=6
Jul 17 00:57:24.367567 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:24.367247    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" event={"ID":"1dacf728-3d0a-45cb-bbc0-bf85419d744b","Type":"ContainerDied","Data":"2a6913201fad6e9b6df177419ab5b72d32b9796dac396813f879cff79feb92d1"}
Jul 17 00:57:24.367567 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:24.367330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" event={"ID":"1dacf728-3d0a-45cb-bbc0-bf85419d744b","Type":"ContainerStarted","Data":"ffafc27926304105d5c7574a4b7f5bfe621c9d55db879a7508414035b496c7a2"}
Jul 17 00:57:24.367567 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:24.367443    2573 scope.go:117] "RemoveContainer" containerID="2a6913201fad6e9b6df177419ab5b72d32b9796dac396813f879cff79feb92d1"
Jul 17 00:57:25.374502 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:25.374474    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerID="f469beb3f19a8bd38a407c4594dcfad56b3599fedeaf110554b6325791ffab91" exitCode=6
Jul 17 00:57:25.374904 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:25.374514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" event={"ID":"1dacf728-3d0a-45cb-bbc0-bf85419d744b","Type":"ContainerDied","Data":"f469beb3f19a8bd38a407c4594dcfad56b3599fedeaf110554b6325791ffab91"}
Jul 17 00:57:25.374904 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:25.374589    2573 scope.go:117] "RemoveContainer" containerID="2a6913201fad6e9b6df177419ab5b72d32b9796dac396813f879cff79feb92d1"
Jul 17 00:57:25.374904 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:25.374749    2573 scope.go:117] "RemoveContainer" containerID="f469beb3f19a8bd38a407c4594dcfad56b3599fedeaf110554b6325791ffab91"
Jul 17 00:57:25.375059 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:25.374951    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn_odh-ai-gateway-infra(1dacf728-3d0a-45cb-bbc0-bf85419d744b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b"
Jul 17 00:57:26.382446 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:26.382419    2573 scope.go:117] "RemoveContainer" containerID="f469beb3f19a8bd38a407c4594dcfad56b3599fedeaf110554b6325791ffab91"
Jul 17 00:57:26.382857 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:26.382576    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn_odh-ai-gateway-infra(1dacf728-3d0a-45cb-bbc0-bf85419d744b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b"
Jul 17 00:57:29.962378 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:29.962345    2573 scope.go:117] "RemoveContainer" containerID="b28140f79b974ddce935a783545dadddd45b946a8b0c4b7105d41d17aebbe192"
Jul 17 00:57:30.405230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.405198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" event={"ID":"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad","Type":"ContainerStarted","Data":"2173dd64f3bd553527f5619e65870c9c2aeeceb741d0b0575e49101545c5c8a1"}
Jul 17 00:57:30.419499 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.419444    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" podStartSLOduration=14.419426739 podStartE2EDuration="14.419426739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:57:30.418756332 +0000 UTC m=+2444.994908906" watchObservedRunningTime="2026-07-17 00:57:30.419426739 +0000 UTC m=+2444.995579314"
Jul 17 00:57:30.605290 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.605256    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"]
Jul 17 00:57:30.608753 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.608733    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"
Jul 17 00:57:30.616367 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.616349    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"]
Jul 17 00:57:30.712210 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.712123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c131f0f-610f-4733-be7c-9ae0b40e45a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml\" (UID: \"0c131f0f-610f-4733-be7c-9ae0b40e45a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"
Jul 17 00:57:30.813474 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.813443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c131f0f-610f-4733-be7c-9ae0b40e45a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml\" (UID: \"0c131f0f-610f-4733-be7c-9ae0b40e45a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"
Jul 17 00:57:30.814058 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.814039    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c131f0f-610f-4733-be7c-9ae0b40e45a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml\" (UID: \"0c131f0f-610f-4733-be7c-9ae0b40e45a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"
Jul 17 00:57:30.921295 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:30.921275    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"
Jul 17 00:57:31.050432 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:57:31.050401    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c131f0f_610f_4733_be7c_9ae0b40e45a6.slice/crio-68a045b804f0471e187981acee8115bd1fdaf466e0adfc9046c066f1fe15d2bb WatchSource:0}: Error finding container 68a045b804f0471e187981acee8115bd1fdaf466e0adfc9046c066f1fe15d2bb: Status 404 returned error can't find the container with id 68a045b804f0471e187981acee8115bd1fdaf466e0adfc9046c066f1fe15d2bb
Jul 17 00:57:31.050841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.050549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"]
Jul 17 00:57:31.411675 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.411640    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerID="2173dd64f3bd553527f5619e65870c9c2aeeceb741d0b0575e49101545c5c8a1" exitCode=6
Jul 17 00:57:31.411838 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.411713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" event={"ID":"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad","Type":"ContainerDied","Data":"2173dd64f3bd553527f5619e65870c9c2aeeceb741d0b0575e49101545c5c8a1"}
Jul 17 00:57:31.411838 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.411770    2573 scope.go:117] "RemoveContainer" containerID="b28140f79b974ddce935a783545dadddd45b946a8b0c4b7105d41d17aebbe192"
Jul 17 00:57:31.412121 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.412101    2573 scope.go:117] "RemoveContainer" containerID="2173dd64f3bd553527f5619e65870c9c2aeeceb741d0b0575e49101545c5c8a1"
Jul 17 00:57:31.412354 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:31.412328    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-j7k7h_odh-ai-gateway-infra(3d741221-1ce0-463f-a4e5-bfa1f0cb29ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad"
Jul 17 00:57:31.413408 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.413380    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" event={"ID":"0c131f0f-610f-4733-be7c-9ae0b40e45a6","Type":"ContainerStarted","Data":"1199a89d8602034ddcd13f118001a63d305872383b9f8497545e02e3718e974a"}
Jul 17 00:57:31.413507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.413423    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" event={"ID":"0c131f0f-610f-4733-be7c-9ae0b40e45a6","Type":"ContainerStarted","Data":"68a045b804f0471e187981acee8115bd1fdaf466e0adfc9046c066f1fe15d2bb"}
Jul 17 00:57:31.432242 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.432220    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"]
Jul 17 00:57:31.440092 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:31.440056    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" podStartSLOduration=1.440043099 podStartE2EDuration="1.440043099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:57:31.438528859 +0000 UTC m=+2446.014681433" watchObservedRunningTime="2026-07-17 00:57:31.440043099 +0000 UTC m=+2446.016195685"
Jul 17 00:57:32.421909 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:32.421878    2573 generic.go:356] "Generic (PLEG): container finished" podID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerID="1199a89d8602034ddcd13f118001a63d305872383b9f8497545e02e3718e974a" exitCode=6
Jul 17 00:57:32.422305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:32.421923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" event={"ID":"0c131f0f-610f-4733-be7c-9ae0b40e45a6","Type":"ContainerDied","Data":"1199a89d8602034ddcd13f118001a63d305872383b9f8497545e02e3718e974a"}
Jul 17 00:57:32.422305 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:32.422137    2573 scope.go:117] "RemoveContainer" containerID="1199a89d8602034ddcd13f118001a63d305872383b9f8497545e02e3718e974a"
Jul 17 00:57:32.554782 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:32.554758    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"
Jul 17 00:57:32.726871 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:32.726808    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d741221-1ce0-463f-a4e5-bfa1f0cb29ad-maas-api-service-ca\") pod \"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad\" (UID: \"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad\") "
Jul 17 00:57:32.727117 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:32.727092    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d741221-1ce0-463f-a4e5-bfa1f0cb29ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" (UID: "3d741221-1ce0-463f-a4e5-bfa1f0cb29ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:57:32.828293 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:32.828269    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d741221-1ce0-463f-a4e5-bfa1f0cb29ad-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:57:33.428977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.428913    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"
Jul 17 00:57:33.428977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.428940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h" event={"ID":"3d741221-1ce0-463f-a4e5-bfa1f0cb29ad","Type":"ContainerDied","Data":"075bec4d61391f9730f3d51cff118148b5383a937bc5b8161af6db5f1d2bbaf3"}
Jul 17 00:57:33.429433 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.429002    2573 scope.go:117] "RemoveContainer" containerID="2173dd64f3bd553527f5619e65870c9c2aeeceb741d0b0575e49101545c5c8a1"
Jul 17 00:57:33.430971 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.430944    2573 generic.go:356] "Generic (PLEG): container finished" podID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerID="98d9570526d2d81da58a518c3e24a8d3ab48a67d6ebe511c45fa08c78f57dc6a" exitCode=6
Jul 17 00:57:33.431086 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.430987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" event={"ID":"0c131f0f-610f-4733-be7c-9ae0b40e45a6","Type":"ContainerDied","Data":"98d9570526d2d81da58a518c3e24a8d3ab48a67d6ebe511c45fa08c78f57dc6a"}
Jul 17 00:57:33.431214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.431198    2573 scope.go:117] "RemoveContainer" containerID="98d9570526d2d81da58a518c3e24a8d3ab48a67d6ebe511c45fa08c78f57dc6a"
Jul 17 00:57:33.431387 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:33.431368    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml_odh-ai-gateway-infra(0c131f0f-610f-4733-be7c-9ae0b40e45a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6"
Jul 17 00:57:33.437925 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.437911    2573 scope.go:117] "RemoveContainer" containerID="1199a89d8602034ddcd13f118001a63d305872383b9f8497545e02e3718e974a"
Jul 17 00:57:33.457940 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.457917    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"]
Jul 17 00:57:33.460755 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.460735    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-j7k7h"]
Jul 17 00:57:33.973848 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:33.973816    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" path="/var/lib/kubelet/pods/3d741221-1ce0-463f-a4e5-bfa1f0cb29ad/volumes"
Jul 17 00:57:34.439734 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:34.439702    2573 scope.go:117] "RemoveContainer" containerID="98d9570526d2d81da58a518c3e24a8d3ab48a67d6ebe511c45fa08c78f57dc6a"
Jul 17 00:57:34.440111 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:34.439889    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml_odh-ai-gateway-infra(0c131f0f-610f-4733-be7c-9ae0b40e45a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6"
Jul 17 00:57:39.962254 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:39.962220    2573 scope.go:117] "RemoveContainer" containerID="f469beb3f19a8bd38a407c4594dcfad56b3599fedeaf110554b6325791ffab91"
Jul 17 00:57:40.474485 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:40.474454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" event={"ID":"1dacf728-3d0a-45cb-bbc0-bf85419d744b","Type":"ContainerStarted","Data":"ca4826eb7d9c64380b133d7a9df8dac1a1eb08b69d75444dff6eab9208ae164e"}
Jul 17 00:57:40.490241 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:40.489431    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" podStartSLOduration=17.489413371 podStartE2EDuration="17.489413371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:57:40.488433212 +0000 UTC m=+2455.064585785" watchObservedRunningTime="2026-07-17 00:57:40.489413371 +0000 UTC m=+2455.065565946"
Jul 17 00:57:41.485189 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:41.485156    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerID="ca4826eb7d9c64380b133d7a9df8dac1a1eb08b69d75444dff6eab9208ae164e" exitCode=6
Jul 17 00:57:41.485705 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:41.485229    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" event={"ID":"1dacf728-3d0a-45cb-bbc0-bf85419d744b","Type":"ContainerDied","Data":"ca4826eb7d9c64380b133d7a9df8dac1a1eb08b69d75444dff6eab9208ae164e"}
Jul 17 00:57:41.485705 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:41.485336    2573 scope.go:117] "RemoveContainer" containerID="f469beb3f19a8bd38a407c4594dcfad56b3599fedeaf110554b6325791ffab91"
Jul 17 00:57:41.485705 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:41.485606    2573 scope.go:117] "RemoveContainer" containerID="ca4826eb7d9c64380b133d7a9df8dac1a1eb08b69d75444dff6eab9208ae164e"
Jul 17 00:57:41.485945 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:41.485822    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn_odh-ai-gateway-infra(1dacf728-3d0a-45cb-bbc0-bf85419d744b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b"
Jul 17 00:57:41.498668 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:41.498649    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"]
Jul 17 00:57:42.624173 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:42.624149    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"
Jul 17 00:57:42.804843 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:42.804810    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dacf728-3d0a-45cb-bbc0-bf85419d744b-maas-api-service-ca\") pod \"1dacf728-3d0a-45cb-bbc0-bf85419d744b\" (UID: \"1dacf728-3d0a-45cb-bbc0-bf85419d744b\") "
Jul 17 00:57:42.805138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:42.805116    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dacf728-3d0a-45cb-bbc0-bf85419d744b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1dacf728-3d0a-45cb-bbc0-bf85419d744b" (UID: "1dacf728-3d0a-45cb-bbc0-bf85419d744b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:57:42.905934 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:42.905911    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dacf728-3d0a-45cb-bbc0-bf85419d744b-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:57:43.499558 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:43.499530    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"
Jul 17 00:57:43.499734 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:43.499576    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn" event={"ID":"1dacf728-3d0a-45cb-bbc0-bf85419d744b","Type":"ContainerDied","Data":"ffafc27926304105d5c7574a4b7f5bfe621c9d55db879a7508414035b496c7a2"}
Jul 17 00:57:43.499734 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:43.499626    2573 scope.go:117] "RemoveContainer" containerID="ca4826eb7d9c64380b133d7a9df8dac1a1eb08b69d75444dff6eab9208ae164e"
Jul 17 00:57:43.524670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:43.524645    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"]
Jul 17 00:57:43.527875 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:43.527852    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-s9xdn"]
Jul 17 00:57:43.968594 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:43.968562    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" path="/var/lib/kubelet/pods/1dacf728-3d0a-45cb-bbc0-bf85419d744b/volumes"
Jul 17 00:57:46.962264 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:46.962228    2573 scope.go:117] "RemoveContainer" containerID="98d9570526d2d81da58a518c3e24a8d3ab48a67d6ebe511c45fa08c78f57dc6a"
Jul 17 00:57:47.524209 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:47.524175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" event={"ID":"0c131f0f-610f-4733-be7c-9ae0b40e45a6","Type":"ContainerStarted","Data":"9be11c1650afd4e6b51fd0879e7bc59f61e0eb781fd30e908ffa7c9fb6f63b47"}
Jul 17 00:57:48.531118 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.531087    2573 generic.go:356] "Generic (PLEG): container finished" podID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerID="9be11c1650afd4e6b51fd0879e7bc59f61e0eb781fd30e908ffa7c9fb6f63b47" exitCode=6
Jul 17 00:57:48.531605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.531163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" event={"ID":"0c131f0f-610f-4733-be7c-9ae0b40e45a6","Type":"ContainerDied","Data":"9be11c1650afd4e6b51fd0879e7bc59f61e0eb781fd30e908ffa7c9fb6f63b47"}
Jul 17 00:57:48.531605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.531226    2573 scope.go:117] "RemoveContainer" containerID="98d9570526d2d81da58a518c3e24a8d3ab48a67d6ebe511c45fa08c78f57dc6a"
Jul 17 00:57:48.531605 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.531551    2573 scope.go:117] "RemoveContainer" containerID="9be11c1650afd4e6b51fd0879e7bc59f61e0eb781fd30e908ffa7c9fb6f63b47"
Jul 17 00:57:48.531808 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:48.531766    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml_odh-ai-gateway-infra(0c131f0f-610f-4733-be7c-9ae0b40e45a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6"
Jul 17 00:57:48.546861 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.546838    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"]
Jul 17 00:57:48.780584 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.780550    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"]
Jul 17 00:57:48.780985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.780968    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.780987    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781001    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781009    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781020    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781028    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781045    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781055    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781073    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781087 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781081    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781560 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781175    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:48.781560 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781188    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:48.781560 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781199    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781560 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781207    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.781560 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.781217    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dacf728-3d0a-45cb-bbc0-bf85419d744b" containerName="revoke-keys"
Jul 17 00:57:48.785564 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.785543    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"
Jul 17 00:57:48.790562 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.790540    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"]
Jul 17 00:57:48.856039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.856013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-zvz87\" (UID: \"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"
Jul 17 00:57:48.956770 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.956742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-zvz87\" (UID: \"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"
Jul 17 00:57:48.957392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:48.957371    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-zvz87\" (UID: \"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"
Jul 17 00:57:49.098137 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:49.098063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"
Jul 17 00:57:49.230530 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:49.230502    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"]
Jul 17 00:57:49.231587 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:57:49.231556    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f5ea72_b4e8_474e_88a8_f0ed68ad4b8e.slice/crio-0614370b2f4776d9e7e898d1843fdc44dccb60084ce010ab2404d445a42ec8aa WatchSource:0}: Error finding container 0614370b2f4776d9e7e898d1843fdc44dccb60084ce010ab2404d445a42ec8aa: Status 404 returned error can't find the container with id 0614370b2f4776d9e7e898d1843fdc44dccb60084ce010ab2404d445a42ec8aa
Jul 17 00:57:49.538063 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:49.538036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" event={"ID":"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e","Type":"ContainerStarted","Data":"0614370b2f4776d9e7e898d1843fdc44dccb60084ce010ab2404d445a42ec8aa"}
Jul 17 00:57:49.670418 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:49.670394    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"
Jul 17 00:57:49.761985 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:49.761958    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c131f0f-610f-4733-be7c-9ae0b40e45a6-maas-api-service-ca\") pod \"0c131f0f-610f-4733-be7c-9ae0b40e45a6\" (UID: \"0c131f0f-610f-4733-be7c-9ae0b40e45a6\") "
Jul 17 00:57:49.762273 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:49.762254    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c131f0f-610f-4733-be7c-9ae0b40e45a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c131f0f-610f-4733-be7c-9ae0b40e45a6" (UID: "0c131f0f-610f-4733-be7c-9ae0b40e45a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:57:49.863174 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:49.863151    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c131f0f-610f-4733-be7c-9ae0b40e45a6-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:57:50.546734 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.546701    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerID="5eb782e998e8d634e646f223e03c88d72891a38f7cdf7472fab56a8dc418a9b2" exitCode=6
Jul 17 00:57:50.547132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.546808    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" event={"ID":"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e","Type":"ContainerDied","Data":"5eb782e998e8d634e646f223e03c88d72891a38f7cdf7472fab56a8dc418a9b2"}
Jul 17 00:57:50.547132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.546975    2573 scope.go:117] "RemoveContainer" containerID="5eb782e998e8d634e646f223e03c88d72891a38f7cdf7472fab56a8dc418a9b2"
Jul 17 00:57:50.548279 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.548258    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml" event={"ID":"0c131f0f-610f-4733-be7c-9ae0b40e45a6","Type":"ContainerDied","Data":"68a045b804f0471e187981acee8115bd1fdaf466e0adfc9046c066f1fe15d2bb"}
Jul 17 00:57:50.548396 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.548286    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"
Jul 17 00:57:50.548396 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.548311    2573 scope.go:117] "RemoveContainer" containerID="9be11c1650afd4e6b51fd0879e7bc59f61e0eb781fd30e908ffa7c9fb6f63b47"
Jul 17 00:57:50.573934 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.573910    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"]
Jul 17 00:57:50.576844 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:50.576826    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-v8nml"]
Jul 17 00:57:51.556407 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:51.556378    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerID="f9d49148c355355d5ef7f478299286e692d3dd2e245d1536a45ee949422e0588" exitCode=6
Jul 17 00:57:51.556863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:51.556467    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" event={"ID":"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e","Type":"ContainerDied","Data":"f9d49148c355355d5ef7f478299286e692d3dd2e245d1536a45ee949422e0588"}
Jul 17 00:57:51.556863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:51.556526    2573 scope.go:117] "RemoveContainer" containerID="5eb782e998e8d634e646f223e03c88d72891a38f7cdf7472fab56a8dc418a9b2"
Jul 17 00:57:51.556863 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:51.556664    2573 scope.go:117] "RemoveContainer" containerID="f9d49148c355355d5ef7f478299286e692d3dd2e245d1536a45ee949422e0588"
Jul 17 00:57:51.557029 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:51.556908    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-zvz87_odh-ai-gateway-infra(e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e"
Jul 17 00:57:51.968873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:51.968774    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" path="/var/lib/kubelet/pods/0c131f0f-610f-4733-be7c-9ae0b40e45a6/volumes"
Jul 17 00:57:52.565690 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:52.565663    2573 scope.go:117] "RemoveContainer" containerID="f9d49148c355355d5ef7f478299286e692d3dd2e245d1536a45ee949422e0588"
Jul 17 00:57:52.566178 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:57:52.565946    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-zvz87_odh-ai-gateway-infra(e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e"
Jul 17 00:57:54.997073 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997038    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"]
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997408    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997420    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997431    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997436    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997444    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997449    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997458    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997464    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997518    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:54.997524 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997526    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d741221-1ce0-463f-a4e5-bfa1f0cb29ad" containerName="revoke-keys"
Jul 17 00:57:54.997955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:54.997535    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:57:55.000584 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.000566    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.002554 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.002522    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-380fc75a-serving-cert\""
Jul 17 00:57:55.002654 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.002557    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 00:57:55.009277 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.009250    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"]
Jul 17 00:57:55.103472 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.103443    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9wq\" (UniqueName: \"kubernetes.io/projected/7fa8b251-fba0-48b3-b493-828fc83b8d5d-kube-api-access-4p9wq\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.103614 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.103536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa8b251-fba0-48b3-b493-828fc83b8d5d-maas-api-tls\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.103614 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.103555    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8b251-fba0-48b3-b493-828fc83b8d5d-openshift-service-ca\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.204427 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.204397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa8b251-fba0-48b3-b493-828fc83b8d5d-maas-api-tls\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.204546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.204435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8b251-fba0-48b3-b493-828fc83b8d5d-openshift-service-ca\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.204546 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.204465    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9wq\" (UniqueName: \"kubernetes.io/projected/7fa8b251-fba0-48b3-b493-828fc83b8d5d-kube-api-access-4p9wq\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.205047 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.205028    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8b251-fba0-48b3-b493-828fc83b8d5d-openshift-service-ca\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.206892 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.206871    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa8b251-fba0-48b3-b493-828fc83b8d5d-maas-api-tls\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.212762 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.212735    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9wq\" (UniqueName: \"kubernetes.io/projected/7fa8b251-fba0-48b3-b493-828fc83b8d5d-kube-api-access-4p9wq\") pod \"maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.312109 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.312080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:55.448335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.448309    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"]
Jul 17 00:57:55.450971 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:57:55.450935    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa8b251_fba0_48b3_b493_828fc83b8d5d.slice/crio-58c8ff9427931a2e52c6020a26b3f9877ccc00f28dfcc44932888cfff5471181 WatchSource:0}: Error finding container 58c8ff9427931a2e52c6020a26b3f9877ccc00f28dfcc44932888cfff5471181: Status 404 returned error can't find the container with id 58c8ff9427931a2e52c6020a26b3f9877ccc00f28dfcc44932888cfff5471181
Jul 17 00:57:55.583773 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:55.583704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5" event={"ID":"7fa8b251-fba0-48b3-b493-828fc83b8d5d","Type":"ContainerStarted","Data":"58c8ff9427931a2e52c6020a26b3f9877ccc00f28dfcc44932888cfff5471181"}
Jul 17 00:57:56.593132 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:56.593058    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5" event={"ID":"7fa8b251-fba0-48b3-b493-828fc83b8d5d","Type":"ContainerStarted","Data":"dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e"}
Jul 17 00:57:56.593487 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:56.593144    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:57:56.609011 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:57:56.608967    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5" podStartSLOduration=2.239333556 podStartE2EDuration="2.608953449s" totalImagesPullingTime="369.619893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:57:56.606514406 +0000 UTC m=+2471.182666979" watchObservedRunningTime="2026-07-17 00:57:56.608953449 +0000 UTC m=+2471.185106049"
Jul 17 00:58:02.604836 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:02.604806    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:58:06.156878 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.156830    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"]
Jul 17 00:58:06.157407 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.157387    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c131f0f-610f-4733-be7c-9ae0b40e45a6" containerName="revoke-keys"
Jul 17 00:58:06.161821 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.161779    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"
Jul 17 00:58:06.167510 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.167485    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"]
Jul 17 00:58:06.196458 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.196422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3d032-9da2-4849-924f-0a2f9a4ccb18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-xbqcn\" (UID: \"8dc3d032-9da2-4849-924f-0a2f9a4ccb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"
Jul 17 00:58:06.297230 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.297198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3d032-9da2-4849-924f-0a2f9a4ccb18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-xbqcn\" (UID: \"8dc3d032-9da2-4849-924f-0a2f9a4ccb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"
Jul 17 00:58:06.297873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.297839    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3d032-9da2-4849-924f-0a2f9a4ccb18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-xbqcn\" (UID: \"8dc3d032-9da2-4849-924f-0a2f9a4ccb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"
Jul 17 00:58:06.444766 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.444692    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"]
Jul 17 00:58:06.448123 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.448101    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"
Jul 17 00:58:06.466957 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.466899    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"]
Jul 17 00:58:06.474060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.474039    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"
Jul 17 00:58:06.499098 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.499072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3fad2-2590-453a-9713-392df9302bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-380fc75a-8prlc\" (UID: \"0ba3fad2-2590-453a-9713-392df9302bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"
Jul 17 00:58:06.599907 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.599876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3fad2-2590-453a-9713-392df9302bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-380fc75a-8prlc\" (UID: \"0ba3fad2-2590-453a-9713-392df9302bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"
Jul 17 00:58:06.600459 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.600440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3fad2-2590-453a-9713-392df9302bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-380fc75a-8prlc\" (UID: \"0ba3fad2-2590-453a-9713-392df9302bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"
Jul 17 00:58:06.608495 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.608463    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"]
Jul 17 00:58:06.609868 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:58:06.609847    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc3d032_9da2_4849_924f_0a2f9a4ccb18.slice/crio-611c7d7f3cbec66c92819312c718fc738a54a4569cd4fdad853e3c2f29da8975 WatchSource:0}: Error finding container 611c7d7f3cbec66c92819312c718fc738a54a4569cd4fdad853e3c2f29da8975: Status 404 returned error can't find the container with id 611c7d7f3cbec66c92819312c718fc738a54a4569cd4fdad853e3c2f29da8975
Jul 17 00:58:06.648849 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.648817    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" event={"ID":"8dc3d032-9da2-4849-924f-0a2f9a4ccb18","Type":"ContainerStarted","Data":"611c7d7f3cbec66c92819312c718fc738a54a4569cd4fdad853e3c2f29da8975"}
Jul 17 00:58:06.772585 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.772554    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"
Jul 17 00:58:06.916681 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:06.916658    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"]
Jul 17 00:58:06.917588 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:58:06.917561    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba3fad2_2590_453a_9713_392df9302bdf.slice/crio-51d8ce68c558b01eef29bc25fdba2bf2c23c94fc365b6db4eeb5ae2eadadaf1a WatchSource:0}: Error finding container 51d8ce68c558b01eef29bc25fdba2bf2c23c94fc365b6db4eeb5ae2eadadaf1a: Status 404 returned error can't find the container with id 51d8ce68c558b01eef29bc25fdba2bf2c23c94fc365b6db4eeb5ae2eadadaf1a
Jul 17 00:58:07.657141 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:07.657106    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc" event={"ID":"0ba3fad2-2590-453a-9713-392df9302bdf","Type":"ContainerStarted","Data":"b2b7c50814524e63b02f8d2c94d9b0da32f0a97e2890d6dbf1189ca6f11dbf66"}
Jul 17 00:58:07.657521 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:07.657171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc" event={"ID":"0ba3fad2-2590-453a-9713-392df9302bdf","Type":"ContainerStarted","Data":"51d8ce68c558b01eef29bc25fdba2bf2c23c94fc365b6db4eeb5ae2eadadaf1a"}
Jul 17 00:58:07.658936 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:07.658906    2573 generic.go:356] "Generic (PLEG): container finished" podID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerID="426e02eefe863d1e5b1160aed7bad040442ba888015bb4354d5f1b3b47012062" exitCode=6
Jul 17 00:58:07.659042 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:07.658939    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" event={"ID":"8dc3d032-9da2-4849-924f-0a2f9a4ccb18","Type":"ContainerDied","Data":"426e02eefe863d1e5b1160aed7bad040442ba888015bb4354d5f1b3b47012062"}
Jul 17 00:58:07.659144 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:07.659128    2573 scope.go:117] "RemoveContainer" containerID="426e02eefe863d1e5b1160aed7bad040442ba888015bb4354d5f1b3b47012062"
Jul 17 00:58:07.674242 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:07.674194    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc" podStartSLOduration=1.6741758 podStartE2EDuration="1.6741758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:58:07.673688673 +0000 UTC m=+2482.249841247" watchObservedRunningTime="2026-07-17 00:58:07.6741758 +0000 UTC m=+2482.250328385"
Jul 17 00:58:07.963127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:07.963108    2573 scope.go:117] "RemoveContainer" containerID="f9d49148c355355d5ef7f478299286e692d3dd2e245d1536a45ee949422e0588"
Jul 17 00:58:08.666241 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.666212    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerID="269eeaba059b3ebe0537357ce35a828bb1895719c520b86a81874102a433efeb" exitCode=6
Jul 17 00:58:08.666581 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.666285    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" event={"ID":"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e","Type":"ContainerDied","Data":"269eeaba059b3ebe0537357ce35a828bb1895719c520b86a81874102a433efeb"}
Jul 17 00:58:08.666581 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.666342    2573 scope.go:117] "RemoveContainer" containerID="f9d49148c355355d5ef7f478299286e692d3dd2e245d1536a45ee949422e0588"
Jul 17 00:58:08.666682 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.666628    2573 scope.go:117] "RemoveContainer" containerID="269eeaba059b3ebe0537357ce35a828bb1895719c520b86a81874102a433efeb"
Jul 17 00:58:08.666881 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:08.666851    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-zvz87_odh-ai-gateway-infra(e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e"
Jul 17 00:58:08.667893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.667869    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ba3fad2-2590-453a-9713-392df9302bdf" containerID="b2b7c50814524e63b02f8d2c94d9b0da32f0a97e2890d6dbf1189ca6f11dbf66" exitCode=0
Jul 17 00:58:08.667989 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.667950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc" event={"ID":"0ba3fad2-2590-453a-9713-392df9302bdf","Type":"ContainerDied","Data":"b2b7c50814524e63b02f8d2c94d9b0da32f0a97e2890d6dbf1189ca6f11dbf66"}
Jul 17 00:58:08.669695 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.669672    2573 generic.go:356] "Generic (PLEG): container finished" podID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerID="cb99d17d49c0864a84ad4620347217cd933dc8ca194d80b0f1e66452b1622973" exitCode=6
Jul 17 00:58:08.669811 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.669726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" event={"ID":"8dc3d032-9da2-4849-924f-0a2f9a4ccb18","Type":"ContainerDied","Data":"cb99d17d49c0864a84ad4620347217cd933dc8ca194d80b0f1e66452b1622973"}
Jul 17 00:58:08.669994 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.669976    2573 scope.go:117] "RemoveContainer" containerID="cb99d17d49c0864a84ad4620347217cd933dc8ca194d80b0f1e66452b1622973"
Jul 17 00:58:08.670160 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:08.670140    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-xbqcn_odh-ai-gateway-infra(8dc3d032-9da2-4849-924f-0a2f9a4ccb18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18"
Jul 17 00:58:08.703104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:08.703074    2573 scope.go:117] "RemoveContainer" containerID="426e02eefe863d1e5b1160aed7bad040442ba888015bb4354d5f1b3b47012062"
Jul 17 00:58:09.677187 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.677160    2573 scope.go:117] "RemoveContainer" containerID="cb99d17d49c0864a84ad4620347217cd933dc8ca194d80b0f1e66452b1622973"
Jul 17 00:58:09.677622 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:09.677354    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-xbqcn_odh-ai-gateway-infra(8dc3d032-9da2-4849-924f-0a2f9a4ccb18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18"
Jul 17 00:58:09.690512 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.690482    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"]
Jul 17 00:58:09.829478 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.829454    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"
Jul 17 00:58:09.858670 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.858644    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"
Jul 17 00:58:09.923526 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.923499    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3fad2-2590-453a-9713-392df9302bdf-maas-api-service-ca\") pod \"0ba3fad2-2590-453a-9713-392df9302bdf\" (UID: \"0ba3fad2-2590-453a-9713-392df9302bdf\") "
Jul 17 00:58:09.923666 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.923579    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e-maas-api-service-ca\") pod \"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e\" (UID: \"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e\") "
Jul 17 00:58:09.923906 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.923882    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba3fad2-2590-453a-9713-392df9302bdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ba3fad2-2590-453a-9713-392df9302bdf" (UID: "0ba3fad2-2590-453a-9713-392df9302bdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:58:09.923987 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:09.923949    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" (UID: "e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:58:10.024333 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.024299    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3fad2-2590-453a-9713-392df9302bdf-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:10.024333 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.024328    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:10.685387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.685358    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"
Jul 17 00:58:10.685816 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.685386    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87" event={"ID":"e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e","Type":"ContainerDied","Data":"0614370b2f4776d9e7e898d1843fdc44dccb60084ce010ab2404d445a42ec8aa"}
Jul 17 00:58:10.685816 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.685446    2573 scope.go:117] "RemoveContainer" containerID="269eeaba059b3ebe0537357ce35a828bb1895719c520b86a81874102a433efeb"
Jul 17 00:58:10.687011 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.686995    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc"
Jul 17 00:58:10.687143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.686994    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-380fc75a-8prlc" event={"ID":"0ba3fad2-2590-453a-9713-392df9302bdf","Type":"ContainerDied","Data":"51d8ce68c558b01eef29bc25fdba2bf2c23c94fc365b6db4eeb5ae2eadadaf1a"}
Jul 17 00:58:10.687143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.687126    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d8ce68c558b01eef29bc25fdba2bf2c23c94fc365b6db4eeb5ae2eadadaf1a"
Jul 17 00:58:10.711381 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.711359    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"]
Jul 17 00:58:10.721955 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:10.721934    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zvz87"]
Jul 17 00:58:11.968195 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:11.968154    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" path="/var/lib/kubelet/pods/e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e/volumes"
Jul 17 00:58:17.044335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.044272    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"]
Jul 17 00:58:17.044757 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.044542    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5" podUID="7fa8b251-fba0-48b3-b493-828fc83b8d5d" containerName="maas-api" containerID="cri-o://dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e" gracePeriod=30
Jul 17 00:58:17.299276 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.299219    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:58:17.382741 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.382709    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p9wq\" (UniqueName: \"kubernetes.io/projected/7fa8b251-fba0-48b3-b493-828fc83b8d5d-kube-api-access-4p9wq\") pod \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") "
Jul 17 00:58:17.382920 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.382826    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8b251-fba0-48b3-b493-828fc83b8d5d-openshift-service-ca\") pod \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") "
Jul 17 00:58:17.382920 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.382893    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa8b251-fba0-48b3-b493-828fc83b8d5d-maas-api-tls\") pod \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\" (UID: \"7fa8b251-fba0-48b3-b493-828fc83b8d5d\") "
Jul 17 00:58:17.383218 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.383192    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa8b251-fba0-48b3-b493-828fc83b8d5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fa8b251-fba0-48b3-b493-828fc83b8d5d" (UID: "7fa8b251-fba0-48b3-b493-828fc83b8d5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:58:17.384698 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.384675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa8b251-fba0-48b3-b493-828fc83b8d5d-kube-api-access-4p9wq" (OuterVolumeSpecName: "kube-api-access-4p9wq") pod "7fa8b251-fba0-48b3-b493-828fc83b8d5d" (UID: "7fa8b251-fba0-48b3-b493-828fc83b8d5d"). InnerVolumeSpecName "kube-api-access-4p9wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:58:17.384874 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.384853    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa8b251-fba0-48b3-b493-828fc83b8d5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fa8b251-fba0-48b3-b493-828fc83b8d5d" (UID: "7fa8b251-fba0-48b3-b493-828fc83b8d5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:58:17.483435 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.483414    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa8b251-fba0-48b3-b493-828fc83b8d5d-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:17.483540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.483435    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p9wq\" (UniqueName: \"kubernetes.io/projected/7fa8b251-fba0-48b3-b493-828fc83b8d5d-kube-api-access-4p9wq\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:17.483540 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.483447    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa8b251-fba0-48b3-b493-828fc83b8d5d-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:17.727781 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.727698    2573 generic.go:356] "Generic (PLEG): container finished" podID="7fa8b251-fba0-48b3-b493-828fc83b8d5d" containerID="dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e" exitCode=0
Jul 17 00:58:17.727781 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.727750    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"
Jul 17 00:58:17.727992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.727783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5" event={"ID":"7fa8b251-fba0-48b3-b493-828fc83b8d5d","Type":"ContainerDied","Data":"dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e"}
Jul 17 00:58:17.727992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.727859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5" event={"ID":"7fa8b251-fba0-48b3-b493-828fc83b8d5d","Type":"ContainerDied","Data":"58c8ff9427931a2e52c6020a26b3f9877ccc00f28dfcc44932888cfff5471181"}
Jul 17 00:58:17.727992 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.727881    2573 scope.go:117] "RemoveContainer" containerID="dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e"
Jul 17 00:58:17.737002 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.736987    2573 scope.go:117] "RemoveContainer" containerID="dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e"
Jul 17 00:58:17.737265 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:17.737247    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e\": container with ID starting with dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e not found: ID does not exist" containerID="dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e"
Jul 17 00:58:17.737324 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.737281    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e"} err="failed to get container status \"dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e\": rpc error: code = NotFound desc = could not find container \"dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e\": container with ID starting with dac35725c6082778ba61e168ab36197a953af89991274c710d9147ae792dc69e not found: ID does not exist"
Jul 17 00:58:17.748469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.748428    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"]
Jul 17 00:58:17.751441 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.751418    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-380fc75a-5565bd99d4-tdmj5"]
Jul 17 00:58:17.969060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:17.969028    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa8b251-fba0-48b3-b493-828fc83b8d5d" path="/var/lib/kubelet/pods/7fa8b251-fba0-48b3-b493-828fc83b8d5d/volumes"
Jul 17 00:58:20.645374 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645337    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"]
Jul 17 00:58:20.645900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645882    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa8b251-fba0-48b3-b493-828fc83b8d5d" containerName="maas-api"
Jul 17 00:58:20.645961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645902    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa8b251-fba0-48b3-b493-828fc83b8d5d" containerName="maas-api"
Jul 17 00:58:20.645961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645915    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:20.645961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645923    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:20.645961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645933    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba3fad2-2590-453a-9713-392df9302bdf" containerName="revoke-keys"
Jul 17 00:58:20.645961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645943    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba3fad2-2590-453a-9713-392df9302bdf" containerName="revoke-keys"
Jul 17 00:58:20.645961 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645960    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:20.646151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.645968    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:20.646151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.646060    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:20.646151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.646073    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba3fad2-2590-453a-9713-392df9302bdf" containerName="revoke-keys"
Jul 17 00:58:20.646151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.646082    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:20.646151 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.646093    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa8b251-fba0-48b3-b493-828fc83b8d5d" containerName="maas-api"
Jul 17 00:58:20.650842 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.650821    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"
Jul 17 00:58:20.654344 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.654321    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"]
Jul 17 00:58:20.709176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.709148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a847ed19-91ef-429a-a50c-2d0bed20d9f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp\" (UID: \"a847ed19-91ef-429a-a50c-2d0bed20d9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"
Jul 17 00:58:20.809959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.809933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a847ed19-91ef-429a-a50c-2d0bed20d9f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp\" (UID: \"a847ed19-91ef-429a-a50c-2d0bed20d9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"
Jul 17 00:58:20.810464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.810445    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a847ed19-91ef-429a-a50c-2d0bed20d9f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp\" (UID: \"a847ed19-91ef-429a-a50c-2d0bed20d9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"
Jul 17 00:58:20.963727 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:20.963669    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"
Jul 17 00:58:21.296477 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:21.296450    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"]
Jul 17 00:58:21.298133 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:58:21.298111    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda847ed19_91ef_429a_a50c_2d0bed20d9f9.slice/crio-b2fa86e589586f5d05425dfb458c72bfa28ce55bd45594e475b26aff3061fc7a WatchSource:0}: Error finding container b2fa86e589586f5d05425dfb458c72bfa28ce55bd45594e475b26aff3061fc7a: Status 404 returned error can't find the container with id b2fa86e589586f5d05425dfb458c72bfa28ce55bd45594e475b26aff3061fc7a
Jul 17 00:58:21.752129 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:21.752074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" event={"ID":"a847ed19-91ef-429a-a50c-2d0bed20d9f9","Type":"ContainerStarted","Data":"f6cf58263d4224d6fe5bf1402f5eb2a22e8ba7ebf5b3e3bd6ec94aaa204b63bc"}
Jul 17 00:58:21.752508 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:21.752154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" event={"ID":"a847ed19-91ef-429a-a50c-2d0bed20d9f9","Type":"ContainerStarted","Data":"b2fa86e589586f5d05425dfb458c72bfa28ce55bd45594e475b26aff3061fc7a"}
Jul 17 00:58:21.765575 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:21.765520    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" podStartSLOduration=1.765501055 podStartE2EDuration="1.765501055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:58:21.764574223 +0000 UTC m=+2496.340726797" watchObservedRunningTime="2026-07-17 00:58:21.765501055 +0000 UTC m=+2496.341653630"
Jul 17 00:58:22.758824 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:22.758774    2573 generic.go:356] "Generic (PLEG): container finished" podID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerID="f6cf58263d4224d6fe5bf1402f5eb2a22e8ba7ebf5b3e3bd6ec94aaa204b63bc" exitCode=6
Jul 17 00:58:22.759266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:22.758866    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" event={"ID":"a847ed19-91ef-429a-a50c-2d0bed20d9f9","Type":"ContainerDied","Data":"f6cf58263d4224d6fe5bf1402f5eb2a22e8ba7ebf5b3e3bd6ec94aaa204b63bc"}
Jul 17 00:58:22.759266 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:22.759002    2573 scope.go:117] "RemoveContainer" containerID="f6cf58263d4224d6fe5bf1402f5eb2a22e8ba7ebf5b3e3bd6ec94aaa204b63bc"
Jul 17 00:58:22.962273 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:22.962254    2573 scope.go:117] "RemoveContainer" containerID="cb99d17d49c0864a84ad4620347217cd933dc8ca194d80b0f1e66452b1622973"
Jul 17 00:58:23.086284 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.086223    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"]
Jul 17 00:58:23.086659 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.086641    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:23.086746 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.086663    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:23.086826 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.086751    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f5ea72-b4e8-474e-88a8-f0ed68ad4b8e" containerName="revoke-keys"
Jul 17 00:58:23.090848 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.090830    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"
Jul 17 00:58:23.095329 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.095278    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"]
Jul 17 00:58:23.129137 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.129115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921dc275-28ad-4260-b9f2-e3c90bd156ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7\" (UID: \"921dc275-28ad-4260-b9f2-e3c90bd156ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"
Jul 17 00:58:23.230364 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.230339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921dc275-28ad-4260-b9f2-e3c90bd156ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7\" (UID: \"921dc275-28ad-4260-b9f2-e3c90bd156ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"
Jul 17 00:58:23.230942 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.230921    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921dc275-28ad-4260-b9f2-e3c90bd156ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7\" (UID: \"921dc275-28ad-4260-b9f2-e3c90bd156ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"
Jul 17 00:58:23.402701 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.402648    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"
Jul 17 00:58:23.546486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.546442    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"]
Jul 17 00:58:23.547125 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:58:23.547100    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921dc275_28ad_4260_b9f2_e3c90bd156ce.slice/crio-698fdd244bd10544d41a04fef7ff00a7772f006376d16b34cb045efbea87f0cc WatchSource:0}: Error finding container 698fdd244bd10544d41a04fef7ff00a7772f006376d16b34cb045efbea87f0cc: Status 404 returned error can't find the container with id 698fdd244bd10544d41a04fef7ff00a7772f006376d16b34cb045efbea87f0cc
Jul 17 00:58:23.766346 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.766309    2573 generic.go:356] "Generic (PLEG): container finished" podID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerID="28e713c3aef70cfa05c73a03adc20fd96e68f0c951e9b01a6edd8d73b9516407" exitCode=6
Jul 17 00:58:23.766702 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.766380    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" event={"ID":"8dc3d032-9da2-4849-924f-0a2f9a4ccb18","Type":"ContainerDied","Data":"28e713c3aef70cfa05c73a03adc20fd96e68f0c951e9b01a6edd8d73b9516407"}
Jul 17 00:58:23.766702 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.766441    2573 scope.go:117] "RemoveContainer" containerID="cb99d17d49c0864a84ad4620347217cd933dc8ca194d80b0f1e66452b1622973"
Jul 17 00:58:23.766843 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.766772    2573 scope.go:117] "RemoveContainer" containerID="28e713c3aef70cfa05c73a03adc20fd96e68f0c951e9b01a6edd8d73b9516407"
Jul 17 00:58:23.767029 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:23.767002    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-xbqcn_odh-ai-gateway-infra(8dc3d032-9da2-4849-924f-0a2f9a4ccb18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18"
Jul 17 00:58:23.768247 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.768220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" event={"ID":"921dc275-28ad-4260-b9f2-e3c90bd156ce","Type":"ContainerStarted","Data":"698fdd244bd10544d41a04fef7ff00a7772f006376d16b34cb045efbea87f0cc"}
Jul 17 00:58:23.769994 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.769973    2573 generic.go:356] "Generic (PLEG): container finished" podID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerID="56fd60d22ae6162a2d82c06ae6082874d633fd86f2f4e91f74012a9d06fe00c4" exitCode=6
Jul 17 00:58:23.770093 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.770045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" event={"ID":"a847ed19-91ef-429a-a50c-2d0bed20d9f9","Type":"ContainerDied","Data":"56fd60d22ae6162a2d82c06ae6082874d633fd86f2f4e91f74012a9d06fe00c4"}
Jul 17 00:58:23.770220 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.770208    2573 scope.go:117] "RemoveContainer" containerID="56fd60d22ae6162a2d82c06ae6082874d633fd86f2f4e91f74012a9d06fe00c4"
Jul 17 00:58:23.770364 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:23.770349    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp_odh-ai-gateway-infra(a847ed19-91ef-429a-a50c-2d0bed20d9f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9"
Jul 17 00:58:23.853648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:23.853626    2573 scope.go:117] "RemoveContainer" containerID="f6cf58263d4224d6fe5bf1402f5eb2a22e8ba7ebf5b3e3bd6ec94aaa204b63bc"
Jul 17 00:58:24.776919 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:24.776887    2573 generic.go:356] "Generic (PLEG): container finished" podID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerID="d29957195d1fb5b9dc192f2cce628f7b378ca1a9f40bb0a621552a5d1d4be773" exitCode=6
Jul 17 00:58:24.777315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:24.776971    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" event={"ID":"921dc275-28ad-4260-b9f2-e3c90bd156ce","Type":"ContainerDied","Data":"d29957195d1fb5b9dc192f2cce628f7b378ca1a9f40bb0a621552a5d1d4be773"}
Jul 17 00:58:24.777315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:24.777131    2573 scope.go:117] "RemoveContainer" containerID="d29957195d1fb5b9dc192f2cce628f7b378ca1a9f40bb0a621552a5d1d4be773"
Jul 17 00:58:24.778833 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:24.778816    2573 scope.go:117] "RemoveContainer" containerID="56fd60d22ae6162a2d82c06ae6082874d633fd86f2f4e91f74012a9d06fe00c4"
Jul 17 00:58:24.779004 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:24.778987    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp_odh-ai-gateway-infra(a847ed19-91ef-429a-a50c-2d0bed20d9f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9"
Jul 17 00:58:24.790353 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:24.790336    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"]
Jul 17 00:58:24.925577 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:24.925555    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"
Jul 17 00:58:25.043207 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.043187    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3d032-9da2-4849-924f-0a2f9a4ccb18-maas-api-service-ca\") pod \"8dc3d032-9da2-4849-924f-0a2f9a4ccb18\" (UID: \"8dc3d032-9da2-4849-924f-0a2f9a4ccb18\") "
Jul 17 00:58:25.043541 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.043516    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc3d032-9da2-4849-924f-0a2f9a4ccb18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8dc3d032-9da2-4849-924f-0a2f9a4ccb18" (UID: "8dc3d032-9da2-4849-924f-0a2f9a4ccb18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:58:25.144214 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.144185    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3d032-9da2-4849-924f-0a2f9a4ccb18-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:25.787421 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.787389    2573 generic.go:356] "Generic (PLEG): container finished" podID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerID="5bd7ad666ba54df407f9ef0996f6c4978b7b179f7a8b061b85a71c9f82ee5558" exitCode=6
Jul 17 00:58:25.787879 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.787477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" event={"ID":"921dc275-28ad-4260-b9f2-e3c90bd156ce","Type":"ContainerDied","Data":"5bd7ad666ba54df407f9ef0996f6c4978b7b179f7a8b061b85a71c9f82ee5558"}
Jul 17 00:58:25.787879 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.787531    2573 scope.go:117] "RemoveContainer" containerID="d29957195d1fb5b9dc192f2cce628f7b378ca1a9f40bb0a621552a5d1d4be773"
Jul 17 00:58:25.787879 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.787694    2573 scope.go:117] "RemoveContainer" containerID="5bd7ad666ba54df407f9ef0996f6c4978b7b179f7a8b061b85a71c9f82ee5558"
Jul 17 00:58:25.788036 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:25.787903    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7_odh-ai-gateway-infra(921dc275-28ad-4260-b9f2-e3c90bd156ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce"
Jul 17 00:58:25.788873 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.788854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn" event={"ID":"8dc3d032-9da2-4849-924f-0a2f9a4ccb18","Type":"ContainerDied","Data":"611c7d7f3cbec66c92819312c718fc738a54a4569cd4fdad853e3c2f29da8975"}
Jul 17 00:58:25.788966 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.788893    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"
Jul 17 00:58:25.798738 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.798723    2573 scope.go:117] "RemoveContainer" containerID="28e713c3aef70cfa05c73a03adc20fd96e68f0c951e9b01a6edd8d73b9516407"
Jul 17 00:58:25.813152 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.813101    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"]
Jul 17 00:58:25.815479 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.815459    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xbqcn"]
Jul 17 00:58:25.969019 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:25.968995    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" path="/var/lib/kubelet/pods/8dc3d032-9da2-4849-924f-0a2f9a4ccb18/volumes"
Jul 17 00:58:26.797887 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:26.797854    2573 scope.go:117] "RemoveContainer" containerID="5bd7ad666ba54df407f9ef0996f6c4978b7b179f7a8b061b85a71c9f82ee5558"
Jul 17 00:58:26.798271 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:26.798061    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7_odh-ai-gateway-infra(921dc275-28ad-4260-b9f2-e3c90bd156ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce"
Jul 17 00:58:36.962212 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:36.962176    2573 scope.go:117] "RemoveContainer" containerID="56fd60d22ae6162a2d82c06ae6082874d633fd86f2f4e91f74012a9d06fe00c4"
Jul 17 00:58:37.857507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:37.857439    2573 generic.go:356] "Generic (PLEG): container finished" podID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerID="46d40201c50265f2035314948f3f2b7b94c2fce8e8322795216c1828a83a0a43" exitCode=6
Jul 17 00:58:37.857507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:37.857502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" event={"ID":"a847ed19-91ef-429a-a50c-2d0bed20d9f9","Type":"ContainerDied","Data":"46d40201c50265f2035314948f3f2b7b94c2fce8e8322795216c1828a83a0a43"}
Jul 17 00:58:37.857687 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:37.857573    2573 scope.go:117] "RemoveContainer" containerID="56fd60d22ae6162a2d82c06ae6082874d633fd86f2f4e91f74012a9d06fe00c4"
Jul 17 00:58:37.857893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:37.857867    2573 scope.go:117] "RemoveContainer" containerID="46d40201c50265f2035314948f3f2b7b94c2fce8e8322795216c1828a83a0a43"
Jul 17 00:58:37.858087 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:37.858069    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp_odh-ai-gateway-infra(a847ed19-91ef-429a-a50c-2d0bed20d9f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9"
Jul 17 00:58:38.880497 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:38.880470    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"]
Jul 17 00:58:39.017329 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.017307    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"
Jul 17 00:58:39.152002 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.151926    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a847ed19-91ef-429a-a50c-2d0bed20d9f9-maas-api-service-ca\") pod \"a847ed19-91ef-429a-a50c-2d0bed20d9f9\" (UID: \"a847ed19-91ef-429a-a50c-2d0bed20d9f9\") "
Jul 17 00:58:39.152277 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.152251    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a847ed19-91ef-429a-a50c-2d0bed20d9f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a847ed19-91ef-429a-a50c-2d0bed20d9f9" (UID: "a847ed19-91ef-429a-a50c-2d0bed20d9f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:58:39.253309 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.253282    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a847ed19-91ef-429a-a50c-2d0bed20d9f9-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:39.872373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.872348    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"
Jul 17 00:58:39.872534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.872394    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp" event={"ID":"a847ed19-91ef-429a-a50c-2d0bed20d9f9","Type":"ContainerDied","Data":"b2fa86e589586f5d05425dfb458c72bfa28ce55bd45594e475b26aff3061fc7a"}
Jul 17 00:58:39.872534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.872452    2573 scope.go:117] "RemoveContainer" containerID="46d40201c50265f2035314948f3f2b7b94c2fce8e8322795216c1828a83a0a43"
Jul 17 00:58:39.894653 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.894627    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"]
Jul 17 00:58:39.900483 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.900459    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-rp9sp"]
Jul 17 00:58:39.968387 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:39.968359    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" path="/var/lib/kubelet/pods/a847ed19-91ef-429a-a50c-2d0bed20d9f9/volumes"
Jul 17 00:58:40.962363 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:40.962335    2573 scope.go:117] "RemoveContainer" containerID="5bd7ad666ba54df407f9ef0996f6c4978b7b179f7a8b061b85a71c9f82ee5558"
Jul 17 00:58:41.892938 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:41.892863    2573 generic.go:356] "Generic (PLEG): container finished" podID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerID="2c43e58e806b96ccde66c35a6ca5215badda35c57f3c73efd510b133a1054207" exitCode=6
Jul 17 00:58:41.893069 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:41.892936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" event={"ID":"921dc275-28ad-4260-b9f2-e3c90bd156ce","Type":"ContainerDied","Data":"2c43e58e806b96ccde66c35a6ca5215badda35c57f3c73efd510b133a1054207"}
Jul 17 00:58:41.893069 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:41.893017    2573 scope.go:117] "RemoveContainer" containerID="5bd7ad666ba54df407f9ef0996f6c4978b7b179f7a8b061b85a71c9f82ee5558"
Jul 17 00:58:41.893268 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:41.893253    2573 scope.go:117] "RemoveContainer" containerID="2c43e58e806b96ccde66c35a6ca5215badda35c57f3c73efd510b133a1054207"
Jul 17 00:58:41.893460 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:41.893442    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7_odh-ai-gateway-infra(921dc275-28ad-4260-b9f2-e3c90bd156ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce"
Jul 17 00:58:42.918394 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:42.918363    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"]
Jul 17 00:58:43.049903 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.049871    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"
Jul 17 00:58:43.183092 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.183023    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921dc275-28ad-4260-b9f2-e3c90bd156ce-maas-api-service-ca\") pod \"921dc275-28ad-4260-b9f2-e3c90bd156ce\" (UID: \"921dc275-28ad-4260-b9f2-e3c90bd156ce\") "
Jul 17 00:58:43.183399 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.183367    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921dc275-28ad-4260-b9f2-e3c90bd156ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "921dc275-28ad-4260-b9f2-e3c90bd156ce" (UID: "921dc275-28ad-4260-b9f2-e3c90bd156ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:58:43.284028 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.284006    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921dc275-28ad-4260-b9f2-e3c90bd156ce-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:58:43.907097 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.907064    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"
Jul 17 00:58:43.907273 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.907093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7" event={"ID":"921dc275-28ad-4260-b9f2-e3c90bd156ce","Type":"ContainerDied","Data":"698fdd244bd10544d41a04fef7ff00a7772f006376d16b34cb045efbea87f0cc"}
Jul 17 00:58:43.907273 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.907164    2573 scope.go:117] "RemoveContainer" containerID="2c43e58e806b96ccde66c35a6ca5215badda35c57f3c73efd510b133a1054207"
Jul 17 00:58:43.929385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.929361    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"]
Jul 17 00:58:43.934909 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.934886    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-z68z7"]
Jul 17 00:58:43.968257 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:43.968233    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" path="/var/lib/kubelet/pods/921dc275-28ad-4260-b9f2-e3c90bd156ce/volumes"
Jul 17 00:58:48.822500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822464    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"]
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822820    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822832    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822843    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822849    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822860    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822867    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822873    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822878    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822884    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822889    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:48.822893 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822895    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822901    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822981    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822989    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.822995    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.823001    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.823013    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.823020    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:48.823251 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.823026    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:48.827452 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.827434    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"
Jul 17 00:58:48.829315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.829291    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 00:58:48.831460 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.831434    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"]
Jul 17 00:58:48.930617 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:48.930588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b40211-418a-4040-9fda-1c348b25eedd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-vchrc\" (UID: \"c9b40211-418a-4040-9fda-1c348b25eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"
Jul 17 00:58:49.031203 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:49.031178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b40211-418a-4040-9fda-1c348b25eedd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-vchrc\" (UID: \"c9b40211-418a-4040-9fda-1c348b25eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"
Jul 17 00:58:49.031763 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:49.031740    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b40211-418a-4040-9fda-1c348b25eedd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-vchrc\" (UID: \"c9b40211-418a-4040-9fda-1c348b25eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"
Jul 17 00:58:49.140000 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:49.139940    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"
Jul 17 00:58:49.477509 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:49.477486    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"]
Jul 17 00:58:49.478801 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:58:49.478759    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b40211_418a_4040_9fda_1c348b25eedd.slice/crio-a93845fbd8f0eefbd3314a002b49026c3abac175bb61a72a15bb0d262d3aa064 WatchSource:0}: Error finding container a93845fbd8f0eefbd3314a002b49026c3abac175bb61a72a15bb0d262d3aa064: Status 404 returned error can't find the container with id a93845fbd8f0eefbd3314a002b49026c3abac175bb61a72a15bb0d262d3aa064
Jul 17 00:58:49.940597 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:49.940567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" event={"ID":"c9b40211-418a-4040-9fda-1c348b25eedd","Type":"ContainerStarted","Data":"998fdf0f779ce73e0ec52f80bec85d1b2426d1becea68fddfc994c11f8da4214"}
Jul 17 00:58:49.940963 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:49.940620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" event={"ID":"c9b40211-418a-4040-9fda-1c348b25eedd","Type":"ContainerStarted","Data":"a93845fbd8f0eefbd3314a002b49026c3abac175bb61a72a15bb0d262d3aa064"}
Jul 17 00:58:49.955469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:49.955410    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" podStartSLOduration=1.955390889 podStartE2EDuration="1.955390889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:58:49.953105049 +0000 UTC m=+2524.529257627" watchObservedRunningTime="2026-07-17 00:58:49.955390889 +0000 UTC m=+2524.531543464"
Jul 17 00:58:50.947100 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:50.947065    2573 generic.go:356] "Generic (PLEG): container finished" podID="c9b40211-418a-4040-9fda-1c348b25eedd" containerID="998fdf0f779ce73e0ec52f80bec85d1b2426d1becea68fddfc994c11f8da4214" exitCode=6
Jul 17 00:58:50.947532 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:50.947158    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" event={"ID":"c9b40211-418a-4040-9fda-1c348b25eedd","Type":"ContainerDied","Data":"998fdf0f779ce73e0ec52f80bec85d1b2426d1becea68fddfc994c11f8da4214"}
Jul 17 00:58:50.947532 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:50.947314    2573 scope.go:117] "RemoveContainer" containerID="998fdf0f779ce73e0ec52f80bec85d1b2426d1becea68fddfc994c11f8da4214"
Jul 17 00:58:51.954607 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:51.954534    2573 generic.go:356] "Generic (PLEG): container finished" podID="c9b40211-418a-4040-9fda-1c348b25eedd" containerID="6c1c4af74e8af655e697ef4f2b0034058dde163752118efe6695015183ad25ac" exitCode=6
Jul 17 00:58:51.955010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:51.954615    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" event={"ID":"c9b40211-418a-4040-9fda-1c348b25eedd","Type":"ContainerDied","Data":"6c1c4af74e8af655e697ef4f2b0034058dde163752118efe6695015183ad25ac"}
Jul 17 00:58:51.955010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:51.954681    2573 scope.go:117] "RemoveContainer" containerID="998fdf0f779ce73e0ec52f80bec85d1b2426d1becea68fddfc994c11f8da4214"
Jul 17 00:58:51.955010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:51.954914    2573 scope.go:117] "RemoveContainer" containerID="6c1c4af74e8af655e697ef4f2b0034058dde163752118efe6695015183ad25ac"
Jul 17 00:58:51.955160 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:51.955119    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-vchrc_odh-ai-gateway-infra(c9b40211-418a-4040-9fda-1c348b25eedd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" podUID="c9b40211-418a-4040-9fda-1c348b25eedd"
Jul 17 00:58:52.961870 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:52.961846    2573 scope.go:117] "RemoveContainer" containerID="6c1c4af74e8af655e697ef4f2b0034058dde163752118efe6695015183ad25ac"
Jul 17 00:58:52.962244 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:52.962025    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-vchrc_odh-ai-gateway-infra(c9b40211-418a-4040-9fda-1c348b25eedd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" podUID="c9b40211-418a-4040-9fda-1c348b25eedd"
Jul 17 00:58:54.243588 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.243538    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"]
Jul 17 00:58:54.244169 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244148    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:54.244215 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244174    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:54.244215 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244187    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:54.244215 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244196    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:54.244215 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244207    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:54.244350 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244216    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a847ed19-91ef-429a-a50c-2d0bed20d9f9" containerName="revoke-keys"
Jul 17 00:58:54.244350 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244314    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="921dc275-28ad-4260-b9f2-e3c90bd156ce" containerName="revoke-keys"
Jul 17 00:58:54.244350 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.244333    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc3d032-9da2-4849-924f-0a2f9a4ccb18" containerName="revoke-keys"
Jul 17 00:58:54.247404 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.247383    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.249543 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.249496    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 00:58:54.249708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.249684    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0a075aab-serving-cert\""
Jul 17 00:58:54.258160 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.257784    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"]
Jul 17 00:58:54.380380 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.380352    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-maas-api-tls\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.380529 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.380405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wxz\" (UniqueName: \"kubernetes.io/projected/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-kube-api-access-97wxz\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.380529 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.380446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-openshift-service-ca\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.480900 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.480871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-openshift-service-ca\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.481078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.480946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-maas-api-tls\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.481078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.480985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wxz\" (UniqueName: \"kubernetes.io/projected/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-kube-api-access-97wxz\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.481534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.481515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-openshift-service-ca\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.483226 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.483206    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-maas-api-tls\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.488417 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.488393    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wxz\" (UniqueName: \"kubernetes.io/projected/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-kube-api-access-97wxz\") pod \"maas-api-e2e-mt-0a075aab-6767b49db-z8djl\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.562516 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.562495    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:54.691662 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.691637    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"]
Jul 17 00:58:54.693540 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:58:54.693513    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a46c1d3_b20b_4ff8_ae0e_d3c62739536b.slice/crio-fcbdef275f9f5c91073c4a8a7ce31c0731c31ba6813d623aef1b56541f756e65 WatchSource:0}: Error finding container fcbdef275f9f5c91073c4a8a7ce31c0731c31ba6813d623aef1b56541f756e65: Status 404 returned error can't find the container with id fcbdef275f9f5c91073c4a8a7ce31c0731c31ba6813d623aef1b56541f756e65
Jul 17 00:58:54.695256 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.695234    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 00:58:54.975550 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:54.975469    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl" event={"ID":"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b","Type":"ContainerStarted","Data":"fcbdef275f9f5c91073c4a8a7ce31c0731c31ba6813d623aef1b56541f756e65"}
Jul 17 00:58:55.982148 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:55.982105    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl" event={"ID":"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b","Type":"ContainerStarted","Data":"8658290d4dac33fa98d8276f6273c726b931f36cc56e917c3f90216d91edd2f9"}
Jul 17 00:58:55.982507 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:55.982173    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:58:55.997211 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:55.997165    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl" podStartSLOduration=1.480944976 podStartE2EDuration="1.997151815s" totalImagesPullingTime="516.206839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:58:55.995716781 +0000 UTC m=+2530.571869355" watchObservedRunningTime="2026-07-17 00:58:55.997151815 +0000 UTC m=+2530.573304388"
Jul 17 00:58:56.122466 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.122438    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"]
Jul 17 00:58:56.125913 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.125897    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"
Jul 17 00:58:56.131774 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.131749    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"]
Jul 17 00:58:56.298747 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.298717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4d954d-a8d2-48c1-ba77-b3a532c64f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-rmhgd\" (UID: \"db4d954d-a8d2-48c1-ba77-b3a532c64f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"
Jul 17 00:58:56.400010 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.399986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4d954d-a8d2-48c1-ba77-b3a532c64f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-rmhgd\" (UID: \"db4d954d-a8d2-48c1-ba77-b3a532c64f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"
Jul 17 00:58:56.400530 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.400512    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4d954d-a8d2-48c1-ba77-b3a532c64f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-rmhgd\" (UID: \"db4d954d-a8d2-48c1-ba77-b3a532c64f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"
Jul 17 00:58:56.438486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.438455    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"
Jul 17 00:58:56.578536 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.578508    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"]
Jul 17 00:58:56.579880 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:58:56.579852    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d954d_a8d2_48c1_ba77_b3a532c64f30.slice/crio-6a1d0d94c1fcd6fb5f59aadab0adef2558532fdd5753a3c28e1c6c27e994395c WatchSource:0}: Error finding container 6a1d0d94c1fcd6fb5f59aadab0adef2558532fdd5753a3c28e1c6c27e994395c: Status 404 returned error can't find the container with id 6a1d0d94c1fcd6fb5f59aadab0adef2558532fdd5753a3c28e1c6c27e994395c
Jul 17 00:58:56.989838 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.989784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" event={"ID":"db4d954d-a8d2-48c1-ba77-b3a532c64f30","Type":"ContainerStarted","Data":"678b390e671a67fff8e568f3b702724246c7f1aa934798ed46ecc9b41c4bac35"}
Jul 17 00:58:56.990227 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:56.989902    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" event={"ID":"db4d954d-a8d2-48c1-ba77-b3a532c64f30","Type":"ContainerStarted","Data":"6a1d0d94c1fcd6fb5f59aadab0adef2558532fdd5753a3c28e1c6c27e994395c"}
Jul 17 00:58:57.003502 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:57.003453    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" podStartSLOduration=1.00343895 podStartE2EDuration="1.00343895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:58:57.001620556 +0000 UTC m=+2531.577773130" watchObservedRunningTime="2026-07-17 00:58:57.00343895 +0000 UTC m=+2531.579591524"
Jul 17 00:58:57.995901 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:57.995863    2573 generic.go:356] "Generic (PLEG): container finished" podID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerID="678b390e671a67fff8e568f3b702724246c7f1aa934798ed46ecc9b41c4bac35" exitCode=6
Jul 17 00:58:57.996278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:57.995942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" event={"ID":"db4d954d-a8d2-48c1-ba77-b3a532c64f30","Type":"ContainerDied","Data":"678b390e671a67fff8e568f3b702724246c7f1aa934798ed46ecc9b41c4bac35"}
Jul 17 00:58:57.996278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:57.996079    2573 scope.go:117] "RemoveContainer" containerID="678b390e671a67fff8e568f3b702724246c7f1aa934798ed46ecc9b41c4bac35"
Jul 17 00:58:59.003127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:59.003041    2573 generic.go:356] "Generic (PLEG): container finished" podID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerID="b96bdc46c77464ce1ba06cfb6d6a92481ce9a54186aaa25233e987dee662bc67" exitCode=6
Jul 17 00:58:59.003127 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:59.003093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" event={"ID":"db4d954d-a8d2-48c1-ba77-b3a532c64f30","Type":"ContainerDied","Data":"b96bdc46c77464ce1ba06cfb6d6a92481ce9a54186aaa25233e987dee662bc67"}
Jul 17 00:58:59.003573 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:59.003138    2573 scope.go:117] "RemoveContainer" containerID="678b390e671a67fff8e568f3b702724246c7f1aa934798ed46ecc9b41c4bac35"
Jul 17 00:58:59.003573 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:58:59.003320    2573 scope.go:117] "RemoveContainer" containerID="b96bdc46c77464ce1ba06cfb6d6a92481ce9a54186aaa25233e987dee662bc67"
Jul 17 00:58:59.003644 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:58:59.003566    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-rmhgd_odh-ai-gateway-infra(db4d954d-a8d2-48c1-ba77-b3a532c64f30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30"
Jul 17 00:59:00.010648 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:00.010620    2573 scope.go:117] "RemoveContainer" containerID="b96bdc46c77464ce1ba06cfb6d6a92481ce9a54186aaa25233e987dee662bc67"
Jul 17 00:59:00.011089 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:00.010844    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-rmhgd_odh-ai-gateway-infra(db4d954d-a8d2-48c1-ba77-b3a532c64f30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30"
Jul 17 00:59:01.995569 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:01.995538    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:59:03.962945 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:03.962916    2573 scope.go:117] "RemoveContainer" containerID="6c1c4af74e8af655e697ef4f2b0034058dde163752118efe6695015183ad25ac"
Jul 17 00:59:05.038654 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:05.038612    2573 generic.go:356] "Generic (PLEG): container finished" podID="c9b40211-418a-4040-9fda-1c348b25eedd" containerID="9639c756e0f56a1a69615a2017ec0499b2790f13c5e98918fea342e0e9c72a86" exitCode=6
Jul 17 00:59:05.039147 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:05.038687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" event={"ID":"c9b40211-418a-4040-9fda-1c348b25eedd","Type":"ContainerDied","Data":"9639c756e0f56a1a69615a2017ec0499b2790f13c5e98918fea342e0e9c72a86"}
Jul 17 00:59:05.039147 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:05.038758    2573 scope.go:117] "RemoveContainer" containerID="6c1c4af74e8af655e697ef4f2b0034058dde163752118efe6695015183ad25ac"
Jul 17 00:59:05.039147 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:05.039078    2573 scope.go:117] "RemoveContainer" containerID="9639c756e0f56a1a69615a2017ec0499b2790f13c5e98918fea342e0e9c72a86"
Jul 17 00:59:05.039334 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:05.039312    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-vchrc_odh-ai-gateway-infra(c9b40211-418a-4040-9fda-1c348b25eedd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" podUID="c9b40211-418a-4040-9fda-1c348b25eedd"
Jul 17 00:59:06.062262 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:06.062230    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"]
Jul 17 00:59:06.199124 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:06.199101    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"
Jul 17 00:59:06.282603 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:06.282571    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b40211-418a-4040-9fda-1c348b25eedd-maas-api-service-ca\") pod \"c9b40211-418a-4040-9fda-1c348b25eedd\" (UID: \"c9b40211-418a-4040-9fda-1c348b25eedd\") "
Jul 17 00:59:06.282920 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:06.282898    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b40211-418a-4040-9fda-1c348b25eedd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9b40211-418a-4040-9fda-1c348b25eedd" (UID: "c9b40211-418a-4040-9fda-1c348b25eedd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:59:06.383275 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:06.383217    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b40211-418a-4040-9fda-1c348b25eedd-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:07.053620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:07.053589    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc" event={"ID":"c9b40211-418a-4040-9fda-1c348b25eedd","Type":"ContainerDied","Data":"a93845fbd8f0eefbd3314a002b49026c3abac175bb61a72a15bb0d262d3aa064"}
Jul 17 00:59:07.053801 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:07.053644    2573 scope.go:117] "RemoveContainer" containerID="9639c756e0f56a1a69615a2017ec0499b2790f13c5e98918fea342e0e9c72a86"
Jul 17 00:59:07.053801 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:07.053603    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"
Jul 17 00:59:07.074373 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:07.074346    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"]
Jul 17 00:59:07.079940 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:07.079916    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-vchrc"]
Jul 17 00:59:07.967803 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:07.967760    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" path="/var/lib/kubelet/pods/c9b40211-418a-4040-9fda-1c348b25eedd/volumes"
Jul 17 00:59:10.684054 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684024    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"]
Jul 17 00:59:10.684451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684379    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.684451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684390    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.684451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684398    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.684451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684404    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.684451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684414    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.684451 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684420    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.684669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684468    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.684669 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.684478    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:10.688746 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.688728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"
Jul 17 00:59:10.694143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.694118    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"]
Jul 17 00:59:10.719004 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.718979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f264a6-2733-4219-b6dc-937289f1eeb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw\" (UID: \"95f264a6-2733-4219-b6dc-937289f1eeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"
Jul 17 00:59:10.820039 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.820012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f264a6-2733-4219-b6dc-937289f1eeb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw\" (UID: \"95f264a6-2733-4219-b6dc-937289f1eeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"
Jul 17 00:59:10.820589 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.820570    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f264a6-2733-4219-b6dc-937289f1eeb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw\" (UID: \"95f264a6-2733-4219-b6dc-937289f1eeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"
Jul 17 00:59:10.962767 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:10.962712    2573 scope.go:117] "RemoveContainer" containerID="b96bdc46c77464ce1ba06cfb6d6a92481ce9a54186aaa25233e987dee662bc67"
Jul 17 00:59:11.000437 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:11.000416    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"
Jul 17 00:59:11.335436 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:11.335411    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"]
Jul 17 00:59:11.336706 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:59:11.336681    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f264a6_2733_4219_b6dc_937289f1eeb2.slice/crio-2e76b018c146fa3e302d5bfc19c9d4281f1cc21b8a7b53a466c1b6e121403fed WatchSource:0}: Error finding container 2e76b018c146fa3e302d5bfc19c9d4281f1cc21b8a7b53a466c1b6e121403fed: Status 404 returned error can't find the container with id 2e76b018c146fa3e302d5bfc19c9d4281f1cc21b8a7b53a466c1b6e121403fed
Jul 17 00:59:12.084613 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.084586    2573 generic.go:356] "Generic (PLEG): container finished" podID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerID="218b94aba0544f3eef41d5d3eb5cc45d6efc89664f3785a001d1f4d24ccd0e2d" exitCode=6
Jul 17 00:59:12.084965 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.084651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" event={"ID":"db4d954d-a8d2-48c1-ba77-b3a532c64f30","Type":"ContainerDied","Data":"218b94aba0544f3eef41d5d3eb5cc45d6efc89664f3785a001d1f4d24ccd0e2d"}
Jul 17 00:59:12.084965 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.084701    2573 scope.go:117] "RemoveContainer" containerID="b96bdc46c77464ce1ba06cfb6d6a92481ce9a54186aaa25233e987dee662bc67"
Jul 17 00:59:12.085060 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.084974    2573 scope.go:117] "RemoveContainer" containerID="218b94aba0544f3eef41d5d3eb5cc45d6efc89664f3785a001d1f4d24ccd0e2d"
Jul 17 00:59:12.085204 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:12.085183    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-rmhgd_odh-ai-gateway-infra(db4d954d-a8d2-48c1-ba77-b3a532c64f30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30"
Jul 17 00:59:12.086400 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.086381    2573 generic.go:356] "Generic (PLEG): container finished" podID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerID="91bd48ef72f1f223abea77a06815785b882e483a82e50b5d9d24481d3c1c1ea0" exitCode=6
Jul 17 00:59:12.086490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.086431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" event={"ID":"95f264a6-2733-4219-b6dc-937289f1eeb2","Type":"ContainerDied","Data":"91bd48ef72f1f223abea77a06815785b882e483a82e50b5d9d24481d3c1c1ea0"}
Jul 17 00:59:12.086490 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.086465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" event={"ID":"95f264a6-2733-4219-b6dc-937289f1eeb2","Type":"ContainerStarted","Data":"2e76b018c146fa3e302d5bfc19c9d4281f1cc21b8a7b53a466c1b6e121403fed"}
Jul 17 00:59:12.086592 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:12.086581    2573 scope.go:117] "RemoveContainer" containerID="91bd48ef72f1f223abea77a06815785b882e483a82e50b5d9d24481d3c1c1ea0"
Jul 17 00:59:13.095176 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.095141    2573 generic.go:356] "Generic (PLEG): container finished" podID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerID="7a5a839aabab63db093724d864de690bc1971484690912c2a7ca06f6fdd9d68f" exitCode=6
Jul 17 00:59:13.095571 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.095187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" event={"ID":"95f264a6-2733-4219-b6dc-937289f1eeb2","Type":"ContainerDied","Data":"7a5a839aabab63db093724d864de690bc1971484690912c2a7ca06f6fdd9d68f"}
Jul 17 00:59:13.095571 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.095235    2573 scope.go:117] "RemoveContainer" containerID="91bd48ef72f1f223abea77a06815785b882e483a82e50b5d9d24481d3c1c1ea0"
Jul 17 00:59:13.095571 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.095396    2573 scope.go:117] "RemoveContainer" containerID="7a5a839aabab63db093724d864de690bc1971484690912c2a7ca06f6fdd9d68f"
Jul 17 00:59:13.095702 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:13.095584    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw_odh-ai-gateway-infra(95f264a6-2733-4219-b6dc-937289f1eeb2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2"
Jul 17 00:59:13.106481 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.106455    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"]
Jul 17 00:59:13.245851 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.245822    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"
Jul 17 00:59:13.341714 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.341682    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4d954d-a8d2-48c1-ba77-b3a532c64f30-maas-api-service-ca\") pod \"db4d954d-a8d2-48c1-ba77-b3a532c64f30\" (UID: \"db4d954d-a8d2-48c1-ba77-b3a532c64f30\") "
Jul 17 00:59:13.342086 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.342059    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4d954d-a8d2-48c1-ba77-b3a532c64f30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db4d954d-a8d2-48c1-ba77-b3a532c64f30" (UID: "db4d954d-a8d2-48c1-ba77-b3a532c64f30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:59:13.442935 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:13.442847    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4d954d-a8d2-48c1-ba77-b3a532c64f30-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:14.101339 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:14.101311    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"
Jul 17 00:59:14.101845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:14.101360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd" event={"ID":"db4d954d-a8d2-48c1-ba77-b3a532c64f30","Type":"ContainerDied","Data":"6a1d0d94c1fcd6fb5f59aadab0adef2558532fdd5753a3c28e1c6c27e994395c"}
Jul 17 00:59:14.101845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:14.101454    2573 scope.go:117] "RemoveContainer" containerID="218b94aba0544f3eef41d5d3eb5cc45d6efc89664f3785a001d1f4d24ccd0e2d"
Jul 17 00:59:14.103493 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:14.103415    2573 scope.go:117] "RemoveContainer" containerID="7a5a839aabab63db093724d864de690bc1971484690912c2a7ca06f6fdd9d68f"
Jul 17 00:59:14.103633 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:14.103612    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw_odh-ai-gateway-infra(95f264a6-2733-4219-b6dc-937289f1eeb2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2"
Jul 17 00:59:14.117223 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:14.117203    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"]
Jul 17 00:59:14.123035 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:14.123012    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-rmhgd"]
Jul 17 00:59:15.968545 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:15.968514    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" path="/var/lib/kubelet/pods/db4d954d-a8d2-48c1-ba77-b3a532c64f30/volumes"
Jul 17 00:59:23.133240 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133206    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"]
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133582    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133593    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133604    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133610    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133688    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b40211-418a-4040-9fda-1c348b25eedd" containerName="revoke-keys"
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133697    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133704    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:23.133722 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.133710    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:23.136663 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.136647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"
Jul 17 00:59:23.142977 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.142954    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"]
Jul 17 00:59:23.218851 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.218826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486bf6c8-8868-4df7-b74f-39981c2cabf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs\" (UID: \"486bf6c8-8868-4df7-b74f-39981c2cabf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"
Jul 17 00:59:23.319959 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.319931    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486bf6c8-8868-4df7-b74f-39981c2cabf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs\" (UID: \"486bf6c8-8868-4df7-b74f-39981c2cabf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"
Jul 17 00:59:23.320483 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.320461    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486bf6c8-8868-4df7-b74f-39981c2cabf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs\" (UID: \"486bf6c8-8868-4df7-b74f-39981c2cabf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"
Jul 17 00:59:23.449017 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.448958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"
Jul 17 00:59:23.576184 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:23.576159    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"]
Jul 17 00:59:23.577551 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:59:23.577523    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486bf6c8_8868_4df7_b74f_39981c2cabf0.slice/crio-c761a2fc34cf66211bd767b9189e440cbf618491e10014d825152f79239b92d6 WatchSource:0}: Error finding container c761a2fc34cf66211bd767b9189e440cbf618491e10014d825152f79239b92d6: Status 404 returned error can't find the container with id c761a2fc34cf66211bd767b9189e440cbf618491e10014d825152f79239b92d6
Jul 17 00:59:24.159136 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:24.159098    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" event={"ID":"486bf6c8-8868-4df7-b74f-39981c2cabf0","Type":"ContainerStarted","Data":"1220ae595aef681789a79a2ae513850179594409d42fe496b9cd124511a19a31"}
Jul 17 00:59:24.159500 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:24.159151    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" event={"ID":"486bf6c8-8868-4df7-b74f-39981c2cabf0","Type":"ContainerStarted","Data":"c761a2fc34cf66211bd767b9189e440cbf618491e10014d825152f79239b92d6"}
Jul 17 00:59:24.177950 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:24.175982    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" podStartSLOduration=1.175963422 podStartE2EDuration="1.175963422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:59:24.172113168 +0000 UTC m=+2558.748265726" watchObservedRunningTime="2026-07-17 00:59:24.175963422 +0000 UTC m=+2558.752115998"
Jul 17 00:59:25.166253 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:25.166218    2573 generic.go:356] "Generic (PLEG): container finished" podID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerID="1220ae595aef681789a79a2ae513850179594409d42fe496b9cd124511a19a31" exitCode=6
Jul 17 00:59:25.166662 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:25.166307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" event={"ID":"486bf6c8-8868-4df7-b74f-39981c2cabf0","Type":"ContainerDied","Data":"1220ae595aef681789a79a2ae513850179594409d42fe496b9cd124511a19a31"}
Jul 17 00:59:25.166662 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:25.166428    2573 scope.go:117] "RemoveContainer" containerID="1220ae595aef681789a79a2ae513850179594409d42fe496b9cd124511a19a31"
Jul 17 00:59:26.174269 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:26.174231    2573 generic.go:356] "Generic (PLEG): container finished" podID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerID="266d6812399de05ff44862b5c26e49c412f05b7277fde625a1c3818aad0b2421" exitCode=6
Jul 17 00:59:26.174715 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:26.174312    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" event={"ID":"486bf6c8-8868-4df7-b74f-39981c2cabf0","Type":"ContainerDied","Data":"266d6812399de05ff44862b5c26e49c412f05b7277fde625a1c3818aad0b2421"}
Jul 17 00:59:26.174715 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:26.174368    2573 scope.go:117] "RemoveContainer" containerID="1220ae595aef681789a79a2ae513850179594409d42fe496b9cd124511a19a31"
Jul 17 00:59:26.174715 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:26.174541    2573 scope.go:117] "RemoveContainer" containerID="266d6812399de05ff44862b5c26e49c412f05b7277fde625a1c3818aad0b2421"
Jul 17 00:59:26.174939 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:26.174748    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs_odh-ai-gateway-infra(486bf6c8-8868-4df7-b74f-39981c2cabf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0"
Jul 17 00:59:27.181645 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:27.181616    2573 scope.go:117] "RemoveContainer" containerID="266d6812399de05ff44862b5c26e49c412f05b7277fde625a1c3818aad0b2421"
Jul 17 00:59:27.182055 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:27.181768    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs_odh-ai-gateway-infra(486bf6c8-8868-4df7-b74f-39981c2cabf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0"
Jul 17 00:59:28.962845 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:28.962814    2573 scope.go:117] "RemoveContainer" containerID="7a5a839aabab63db093724d864de690bc1971484690912c2a7ca06f6fdd9d68f"
Jul 17 00:59:29.193668 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:29.193640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" event={"ID":"95f264a6-2733-4219-b6dc-937289f1eeb2","Type":"ContainerStarted","Data":"863e87aa4b708a79121e7a1c11708fdf4d7dfa06a39f6df7eb7d5d433c9bffbe"}
Jul 17 00:59:29.207307 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:29.207259    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" podStartSLOduration=19.207245954 podStartE2EDuration="19.207245954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:59:29.206206818 +0000 UTC m=+2563.782359392" watchObservedRunningTime="2026-07-17 00:59:29.207245954 +0000 UTC m=+2563.783398529"
Jul 17 00:59:30.201553 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:30.201511    2573 generic.go:356] "Generic (PLEG): container finished" podID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerID="863e87aa4b708a79121e7a1c11708fdf4d7dfa06a39f6df7eb7d5d433c9bffbe" exitCode=6
Jul 17 00:59:30.201991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:30.201584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" event={"ID":"95f264a6-2733-4219-b6dc-937289f1eeb2","Type":"ContainerDied","Data":"863e87aa4b708a79121e7a1c11708fdf4d7dfa06a39f6df7eb7d5d433c9bffbe"}
Jul 17 00:59:30.201991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:30.201644    2573 scope.go:117] "RemoveContainer" containerID="7a5a839aabab63db093724d864de690bc1971484690912c2a7ca06f6fdd9d68f"
Jul 17 00:59:30.201991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:30.201937    2573 scope.go:117] "RemoveContainer" containerID="863e87aa4b708a79121e7a1c11708fdf4d7dfa06a39f6df7eb7d5d433c9bffbe"
Jul 17 00:59:30.202147 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:30.202125    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw_odh-ai-gateway-infra(95f264a6-2733-4219-b6dc-937289f1eeb2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2"
Jul 17 00:59:30.215736 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:30.215714    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"]
Jul 17 00:59:31.355191 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.355167    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"
Jul 17 00:59:31.488538 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.488461    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f264a6-2733-4219-b6dc-937289f1eeb2-maas-api-service-ca\") pod \"95f264a6-2733-4219-b6dc-937289f1eeb2\" (UID: \"95f264a6-2733-4219-b6dc-937289f1eeb2\") "
Jul 17 00:59:31.488837 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.488809    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f264a6-2733-4219-b6dc-937289f1eeb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95f264a6-2733-4219-b6dc-937289f1eeb2" (UID: "95f264a6-2733-4219-b6dc-937289f1eeb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:59:31.589431 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.589395    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f264a6-2733-4219-b6dc-937289f1eeb2-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:31.658348 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658322    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"]
Jul 17 00:59:31.658711 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658695    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.658819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658714    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.658819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658730    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.658819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658735    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.658819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658745    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:31.658819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658750    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4d954d-a8d2-48c1-ba77-b3a532c64f30" containerName="revoke-keys"
Jul 17 00:59:31.658819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658759    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.658819 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658764    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.659182 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658861    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.659182 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658870    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.659182 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.658881    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" containerName="revoke-keys"
Jul 17 00:59:31.661881 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.661865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"
Jul 17 00:59:31.668078 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.668052    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"]
Jul 17 00:59:31.790469 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.790438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd329768-544c-4e2f-8ce5-b9bb9d00a0e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a075aab-t6jql\" (UID: \"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"
Jul 17 00:59:31.891460 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.891439    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd329768-544c-4e2f-8ce5-b9bb9d00a0e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a075aab-t6jql\" (UID: \"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"
Jul 17 00:59:31.892065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.892047    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd329768-544c-4e2f-8ce5-b9bb9d00a0e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a075aab-t6jql\" (UID: \"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"
Jul 17 00:59:31.973590 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:31.973564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"
Jul 17 00:59:32.110647 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:32.110618    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"]
Jul 17 00:59:32.110910 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:59:32.110888    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd329768_544c_4e2f_8ce5_b9bb9d00a0e7.slice/crio-5e470f866a3639d8f83c68d73f5a57124a49d6d8782414d4bf7974f6c60b37dc WatchSource:0}: Error finding container 5e470f866a3639d8f83c68d73f5a57124a49d6d8782414d4bf7974f6c60b37dc: Status 404 returned error can't find the container with id 5e470f866a3639d8f83c68d73f5a57124a49d6d8782414d4bf7974f6c60b37dc
Jul 17 00:59:32.215201 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:32.215171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql" event={"ID":"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7","Type":"ContainerStarted","Data":"5e470f866a3639d8f83c68d73f5a57124a49d6d8782414d4bf7974f6c60b37dc"}
Jul 17 00:59:32.216376 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:32.216358    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"
Jul 17 00:59:32.216486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:32.216370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw" event={"ID":"95f264a6-2733-4219-b6dc-937289f1eeb2","Type":"ContainerDied","Data":"2e76b018c146fa3e302d5bfc19c9d4281f1cc21b8a7b53a466c1b6e121403fed"}
Jul 17 00:59:32.216486 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:32.216432    2573 scope.go:117] "RemoveContainer" containerID="863e87aa4b708a79121e7a1c11708fdf4d7dfa06a39f6df7eb7d5d433c9bffbe"
Jul 17 00:59:32.232250 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:32.232228    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"]
Jul 17 00:59:32.235315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:32.235296    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-7gnqw"]
Jul 17 00:59:33.223491 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:33.223417    2573 generic.go:356] "Generic (PLEG): container finished" podID="dd329768-544c-4e2f-8ce5-b9bb9d00a0e7" containerID="2d936c02a5ca0db1c6eb55b37ff2086d4500ded02f5481d73c5a291a1b1aaa81" exitCode=0
Jul 17 00:59:33.223895 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:33.223505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql" event={"ID":"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7","Type":"ContainerDied","Data":"2d936c02a5ca0db1c6eb55b37ff2086d4500ded02f5481d73c5a291a1b1aaa81"}
Jul 17 00:59:33.969143 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:33.969104    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95f264a6-2733-4219-b6dc-937289f1eeb2" path="/var/lib/kubelet/pods/95f264a6-2733-4219-b6dc-937289f1eeb2/volumes"
Jul 17 00:59:34.365727 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:34.365703    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"
Jul 17 00:59:34.412487 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:34.412461    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd329768-544c-4e2f-8ce5-b9bb9d00a0e7-maas-api-service-ca\") pod \"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7\" (UID: \"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7\") "
Jul 17 00:59:34.412814 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:34.412767    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd329768-544c-4e2f-8ce5-b9bb9d00a0e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd329768-544c-4e2f-8ce5-b9bb9d00a0e7" (UID: "dd329768-544c-4e2f-8ce5-b9bb9d00a0e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:59:34.513394 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:34.513370    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd329768-544c-4e2f-8ce5-b9bb9d00a0e7-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:35.236281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:35.236248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql" event={"ID":"dd329768-544c-4e2f-8ce5-b9bb9d00a0e7","Type":"ContainerDied","Data":"5e470f866a3639d8f83c68d73f5a57124a49d6d8782414d4bf7974f6c60b37dc"}
Jul 17 00:59:35.236281 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:35.236278    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a075aab-t6jql"
Jul 17 00:59:35.236534 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:35.236297    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e470f866a3639d8f83c68d73f5a57124a49d6d8782414d4bf7974f6c60b37dc"
Jul 17 00:59:38.860464 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:38.860432    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"]
Jul 17 00:59:38.860862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:38.860839    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd329768-544c-4e2f-8ce5-b9bb9d00a0e7" containerName="revoke-keys"
Jul 17 00:59:38.860862 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:38.860854    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd329768-544c-4e2f-8ce5-b9bb9d00a0e7" containerName="revoke-keys"
Jul 17 00:59:38.860951 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:38.860936    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd329768-544c-4e2f-8ce5-b9bb9d00a0e7" containerName="revoke-keys"
Jul 17 00:59:38.863991 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:38.863974    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"
Jul 17 00:59:38.870426 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:38.870401    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"]
Jul 17 00:59:38.951285 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:38.951255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02de81-3d2b-4368-94aa-13f9f33534ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz\" (UID: \"be02de81-3d2b-4368-94aa-13f9f33534ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"
Jul 17 00:59:39.052118 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:39.052096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02de81-3d2b-4368-94aa-13f9f33534ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz\" (UID: \"be02de81-3d2b-4368-94aa-13f9f33534ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"
Jul 17 00:59:39.052684 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:39.052664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02de81-3d2b-4368-94aa-13f9f33534ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz\" (UID: \"be02de81-3d2b-4368-94aa-13f9f33534ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"
Jul 17 00:59:39.175418 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:39.175364    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"
Jul 17 00:59:39.311841 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:59:39.311810    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe02de81_3d2b_4368_94aa_13f9f33534ca.slice/crio-301dcd1d0c1a8a87ada358a3e6e59f57eada41e08d0895ad3aabed1683db90c7 WatchSource:0}: Error finding container 301dcd1d0c1a8a87ada358a3e6e59f57eada41e08d0895ad3aabed1683db90c7: Status 404 returned error can't find the container with id 301dcd1d0c1a8a87ada358a3e6e59f57eada41e08d0895ad3aabed1683db90c7
Jul 17 00:59:39.312068 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:39.312041    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"]
Jul 17 00:59:40.266385 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:40.266348    2573 generic.go:356] "Generic (PLEG): container finished" podID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerID="5453d2ccad55cd382525f2ba84dec8a75ab35fd08eec2619df4d105e57e3ea8d" exitCode=6
Jul 17 00:59:40.266750 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:40.266430    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" event={"ID":"be02de81-3d2b-4368-94aa-13f9f33534ca","Type":"ContainerDied","Data":"5453d2ccad55cd382525f2ba84dec8a75ab35fd08eec2619df4d105e57e3ea8d"}
Jul 17 00:59:40.266750 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:40.266480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" event={"ID":"be02de81-3d2b-4368-94aa-13f9f33534ca","Type":"ContainerStarted","Data":"301dcd1d0c1a8a87ada358a3e6e59f57eada41e08d0895ad3aabed1683db90c7"}
Jul 17 00:59:40.266750 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:40.266578    2573 scope.go:117] "RemoveContainer" containerID="5453d2ccad55cd382525f2ba84dec8a75ab35fd08eec2619df4d105e57e3ea8d"
Jul 17 00:59:41.274254 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:41.274221    2573 generic.go:356] "Generic (PLEG): container finished" podID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerID="54928902211a6ec63f855ea882a382b2f42cc3f1b9ffc7aacd4b2f22d104dd57" exitCode=6
Jul 17 00:59:41.274642 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:41.274292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" event={"ID":"be02de81-3d2b-4368-94aa-13f9f33534ca","Type":"ContainerDied","Data":"54928902211a6ec63f855ea882a382b2f42cc3f1b9ffc7aacd4b2f22d104dd57"}
Jul 17 00:59:41.274642 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:41.274358    2573 scope.go:117] "RemoveContainer" containerID="5453d2ccad55cd382525f2ba84dec8a75ab35fd08eec2619df4d105e57e3ea8d"
Jul 17 00:59:41.274642 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:41.274595    2573 scope.go:117] "RemoveContainer" containerID="54928902211a6ec63f855ea882a382b2f42cc3f1b9ffc7aacd4b2f22d104dd57"
Jul 17 00:59:41.274809 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:41.274778    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz_odh-ai-gateway-infra(be02de81-3d2b-4368-94aa-13f9f33534ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca"
Jul 17 00:59:42.111744 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.111714    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"]
Jul 17 00:59:42.112122 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.112065    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl" podUID="0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" containerName="maas-api" containerID="cri-o://8658290d4dac33fa98d8276f6273c726b931f36cc56e917c3f90216d91edd2f9" gracePeriod=30
Jul 17 00:59:42.284359 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.284199    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" containerID="8658290d4dac33fa98d8276f6273c726b931f36cc56e917c3f90216d91edd2f9" exitCode=0
Jul 17 00:59:42.284359 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.284287    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl" event={"ID":"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b","Type":"ContainerDied","Data":"8658290d4dac33fa98d8276f6273c726b931f36cc56e917c3f90216d91edd2f9"}
Jul 17 00:59:42.289315 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.289110    2573 scope.go:117] "RemoveContainer" containerID="54928902211a6ec63f855ea882a382b2f42cc3f1b9ffc7aacd4b2f22d104dd57"
Jul 17 00:59:42.289431 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:42.289338    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz_odh-ai-gateway-infra(be02de81-3d2b-4368-94aa-13f9f33534ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca"
Jul 17 00:59:42.363283 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.363225    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:59:42.478841 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.478813    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97wxz\" (UniqueName: \"kubernetes.io/projected/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-kube-api-access-97wxz\") pod \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") "
Jul 17 00:59:42.478999 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.478877    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-openshift-service-ca\") pod \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") "
Jul 17 00:59:42.478999 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.478966    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-maas-api-tls\") pod \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\" (UID: \"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b\") "
Jul 17 00:59:42.479254 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.479230    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" (UID: "0a46c1d3-b20b-4ff8-ae0e-d3c62739536b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:59:42.480911 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.480889    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" (UID: "0a46c1d3-b20b-4ff8-ae0e-d3c62739536b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 00:59:42.480911 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.480904    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-kube-api-access-97wxz" (OuterVolumeSpecName: "kube-api-access-97wxz") pod "0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" (UID: "0a46c1d3-b20b-4ff8-ae0e-d3c62739536b"). InnerVolumeSpecName "kube-api-access-97wxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 00:59:42.580034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.580003    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:42.580034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.580028    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wxz\" (UniqueName: \"kubernetes.io/projected/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-kube-api-access-97wxz\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:42.580034 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.580037    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:42.962884 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:42.962859    2573 scope.go:117] "RemoveContainer" containerID="266d6812399de05ff44862b5c26e49c412f05b7277fde625a1c3818aad0b2421"
Jul 17 00:59:43.295225 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:43.295199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" event={"ID":"486bf6c8-8868-4df7-b74f-39981c2cabf0","Type":"ContainerStarted","Data":"1115feac65d7f7b7893c0ccb516df734198f1b6e9219a9b0728218680718a7fa"}
Jul 17 00:59:43.296519 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:43.296496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl" event={"ID":"0a46c1d3-b20b-4ff8-ae0e-d3c62739536b","Type":"ContainerDied","Data":"fcbdef275f9f5c91073c4a8a7ce31c0731c31ba6813d623aef1b56541f756e65"}
Jul 17 00:59:43.296620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:43.296544    2573 scope.go:117] "RemoveContainer" containerID="8658290d4dac33fa98d8276f6273c726b931f36cc56e917c3f90216d91edd2f9"
Jul 17 00:59:43.296620 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:43.296507    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"
Jul 17 00:59:43.319422 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:43.319396    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"]
Jul 17 00:59:43.322542 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:43.322523    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a075aab-6767b49db-z8djl"]
Jul 17 00:59:43.968525 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:43.968455    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" path="/var/lib/kubelet/pods/0a46c1d3-b20b-4ff8-ae0e-d3c62739536b/volumes"
Jul 17 00:59:44.304427 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:44.304398    2573 generic.go:356] "Generic (PLEG): container finished" podID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerID="1115feac65d7f7b7893c0ccb516df734198f1b6e9219a9b0728218680718a7fa" exitCode=6
Jul 17 00:59:44.304804 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:44.304476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" event={"ID":"486bf6c8-8868-4df7-b74f-39981c2cabf0","Type":"ContainerDied","Data":"1115feac65d7f7b7893c0ccb516df734198f1b6e9219a9b0728218680718a7fa"}
Jul 17 00:59:44.304804 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:44.304537    2573 scope.go:117] "RemoveContainer" containerID="266d6812399de05ff44862b5c26e49c412f05b7277fde625a1c3818aad0b2421"
Jul 17 00:59:44.304804 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:44.304767    2573 scope.go:117] "RemoveContainer" containerID="1115feac65d7f7b7893c0ccb516df734198f1b6e9219a9b0728218680718a7fa"
Jul 17 00:59:44.304988 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:44.304968    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs_odh-ai-gateway-infra(486bf6c8-8868-4df7-b74f-39981c2cabf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0"
Jul 17 00:59:44.320225 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:44.320204    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"]
Jul 17 00:59:45.455717 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:45.455695    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"
Jul 17 00:59:45.603887 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:45.603813    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486bf6c8-8868-4df7-b74f-39981c2cabf0-maas-api-service-ca\") pod \"486bf6c8-8868-4df7-b74f-39981c2cabf0\" (UID: \"486bf6c8-8868-4df7-b74f-39981c2cabf0\") "
Jul 17 00:59:45.604138 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:45.604114    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486bf6c8-8868-4df7-b74f-39981c2cabf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "486bf6c8-8868-4df7-b74f-39981c2cabf0" (UID: "486bf6c8-8868-4df7-b74f-39981c2cabf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:59:45.704868 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:45.704841    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486bf6c8-8868-4df7-b74f-39981c2cabf0-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 00:59:46.162421 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162392    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"]
Jul 17 00:59:46.162765 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162752    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.162765 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162766    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.162886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162776    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.162886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162782    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.162886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162812    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.162886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162818    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.162886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162833    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" containerName="maas-api"
Jul 17 00:59:46.162886 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162838    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" containerName="maas-api"
Jul 17 00:59:46.163072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162895    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.163072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162903    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a46c1d3-b20b-4ff8-ae0e-d3c62739536b" containerName="maas-api"
Jul 17 00:59:46.163072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162911    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.163072 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.162917    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" containerName="revoke-keys"
Jul 17 00:59:46.165995 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.165974    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"
Jul 17 00:59:46.172031 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.172005    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"]
Jul 17 00:59:46.308416 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.308383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19fbb-3dda-4687-81f9-cb677a969b19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-khgsz\" (UID: \"6db19fbb-3dda-4687-81f9-cb677a969b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"
Jul 17 00:59:46.318644 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.318619    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"
Jul 17 00:59:46.318773 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.318646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs" event={"ID":"486bf6c8-8868-4df7-b74f-39981c2cabf0","Type":"ContainerDied","Data":"c761a2fc34cf66211bd767b9189e440cbf618491e10014d825152f79239b92d6"}
Jul 17 00:59:46.318773 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.318731    2573 scope.go:117] "RemoveContainer" containerID="1115feac65d7f7b7893c0ccb516df734198f1b6e9219a9b0728218680718a7fa"
Jul 17 00:59:46.332511 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.332490    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"]
Jul 17 00:59:46.336370 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.336348    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-bwgfs"]
Jul 17 00:59:46.409278 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.409254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19fbb-3dda-4687-81f9-cb677a969b19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-khgsz\" (UID: \"6db19fbb-3dda-4687-81f9-cb677a969b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"
Jul 17 00:59:46.409837 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.409820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19fbb-3dda-4687-81f9-cb677a969b19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-khgsz\" (UID: \"6db19fbb-3dda-4687-81f9-cb677a969b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"
Jul 17 00:59:46.478958 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.478930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"
Jul 17 00:59:46.617361 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:46.617295    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"]
Jul 17 00:59:46.620119 ip-10-0-139-41 kubenswrapper[2573]: W0717 00:59:46.620091    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db19fbb_3dda_4687_81f9_cb677a969b19.slice/crio-86940508488b71d7c31bab8ec25e624dfe87404f0fddb4b3e50e21002f1d5ed2 WatchSource:0}: Error finding container 86940508488b71d7c31bab8ec25e624dfe87404f0fddb4b3e50e21002f1d5ed2: Status 404 returned error can't find the container with id 86940508488b71d7c31bab8ec25e624dfe87404f0fddb4b3e50e21002f1d5ed2
Jul 17 00:59:47.326207 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:47.326172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" event={"ID":"6db19fbb-3dda-4687-81f9-cb677a969b19","Type":"ContainerStarted","Data":"13dcce167ac361d204c249f70e6fcd8da468cda074638ebb22632c8bf59c9fdc"}
Jul 17 00:59:47.326392 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:47.326229    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" event={"ID":"6db19fbb-3dda-4687-81f9-cb677a969b19","Type":"ContainerStarted","Data":"86940508488b71d7c31bab8ec25e624dfe87404f0fddb4b3e50e21002f1d5ed2"}
Jul 17 00:59:47.340550 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:47.340489    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" podStartSLOduration=1.340470484 podStartE2EDuration="1.340470484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:59:47.338655542 +0000 UTC m=+2581.914808114" watchObservedRunningTime="2026-07-17 00:59:47.340470484 +0000 UTC m=+2581.916623060"
Jul 17 00:59:47.968448 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:47.968416    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486bf6c8-8868-4df7-b74f-39981c2cabf0" path="/var/lib/kubelet/pods/486bf6c8-8868-4df7-b74f-39981c2cabf0/volumes"
Jul 17 00:59:48.334708 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:48.334676    2573 generic.go:356] "Generic (PLEG): container finished" podID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerID="13dcce167ac361d204c249f70e6fcd8da468cda074638ebb22632c8bf59c9fdc" exitCode=6
Jul 17 00:59:48.334902 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:48.334738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" event={"ID":"6db19fbb-3dda-4687-81f9-cb677a969b19","Type":"ContainerDied","Data":"13dcce167ac361d204c249f70e6fcd8da468cda074638ebb22632c8bf59c9fdc"}
Jul 17 00:59:48.334974 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:48.334919    2573 scope.go:117] "RemoveContainer" containerID="13dcce167ac361d204c249f70e6fcd8da468cda074638ebb22632c8bf59c9fdc"
Jul 17 00:59:49.343065 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:49.343028    2573 generic.go:356] "Generic (PLEG): container finished" podID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerID="3ca2ea76a0d6ae8d620b4b119b5b3a4635c2ffeece9af7387299d47d588ceee3" exitCode=6
Jul 17 00:59:49.343505 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:49.343107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" event={"ID":"6db19fbb-3dda-4687-81f9-cb677a969b19","Type":"ContainerDied","Data":"3ca2ea76a0d6ae8d620b4b119b5b3a4635c2ffeece9af7387299d47d588ceee3"}
Jul 17 00:59:49.343505 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:49.343163    2573 scope.go:117] "RemoveContainer" containerID="13dcce167ac361d204c249f70e6fcd8da468cda074638ebb22632c8bf59c9fdc"
Jul 17 00:59:49.343505 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:49.343322    2573 scope.go:117] "RemoveContainer" containerID="3ca2ea76a0d6ae8d620b4b119b5b3a4635c2ffeece9af7387299d47d588ceee3"
Jul 17 00:59:49.343669 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:49.343522    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-khgsz_odh-ai-gateway-infra(6db19fbb-3dda-4687-81f9-cb677a969b19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19"
Jul 17 00:59:50.350536 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:50.350512    2573 scope.go:117] "RemoveContainer" containerID="3ca2ea76a0d6ae8d620b4b119b5b3a4635c2ffeece9af7387299d47d588ceee3"
Jul 17 00:59:50.350946 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:50.350666    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-khgsz_odh-ai-gateway-infra(6db19fbb-3dda-4687-81f9-cb677a969b19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19"
Jul 17 00:59:56.962958 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:56.962925    2573 scope.go:117] "RemoveContainer" containerID="54928902211a6ec63f855ea882a382b2f42cc3f1b9ffc7aacd4b2f22d104dd57"
Jul 17 00:59:57.390334 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:57.390302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" event={"ID":"be02de81-3d2b-4368-94aa-13f9f33534ca","Type":"ContainerStarted","Data":"6dfcfbff64051fe87966aeb3652a51c44712d897f6ec041ee3d6df309954ef62"}
Jul 17 00:59:57.405094 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:57.405051    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" podStartSLOduration=19.405037834 podStartE2EDuration="19.405037834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:59:57.402721735 +0000 UTC m=+2591.978874322" watchObservedRunningTime="2026-07-17 00:59:57.405037834 +0000 UTC m=+2591.981190407"
Jul 17 00:59:58.397677 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:58.397642    2573 generic.go:356] "Generic (PLEG): container finished" podID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerID="6dfcfbff64051fe87966aeb3652a51c44712d897f6ec041ee3d6df309954ef62" exitCode=6
Jul 17 00:59:58.398104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:58.397708    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" event={"ID":"be02de81-3d2b-4368-94aa-13f9f33534ca","Type":"ContainerDied","Data":"6dfcfbff64051fe87966aeb3652a51c44712d897f6ec041ee3d6df309954ef62"}
Jul 17 00:59:58.398104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:58.397807    2573 scope.go:117] "RemoveContainer" containerID="54928902211a6ec63f855ea882a382b2f42cc3f1b9ffc7aacd4b2f22d104dd57"
Jul 17 00:59:58.398104 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:58.398020    2573 scope.go:117] "RemoveContainer" containerID="6dfcfbff64051fe87966aeb3652a51c44712d897f6ec041ee3d6df309954ef62"
Jul 17 00:59:58.398271 ip-10-0-139-41 kubenswrapper[2573]: E0717 00:59:58.398247    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz_odh-ai-gateway-infra(be02de81-3d2b-4368-94aa-13f9f33534ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca"
Jul 17 00:59:58.412448 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:58.412422    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"]
Jul 17 00:59:59.534412 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:59.534386    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"
Jul 17 00:59:59.714335 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:59.714260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02de81-3d2b-4368-94aa-13f9f33534ca-maas-api-service-ca\") pod \"be02de81-3d2b-4368-94aa-13f9f33534ca\" (UID: \"be02de81-3d2b-4368-94aa-13f9f33534ca\") "
Jul 17 00:59:59.714590 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:59.714569    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be02de81-3d2b-4368-94aa-13f9f33534ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be02de81-3d2b-4368-94aa-13f9f33534ca" (UID: "be02de81-3d2b-4368-94aa-13f9f33534ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 00:59:59.814956 ip-10-0-139-41 kubenswrapper[2573]: I0717 00:59:59.814926    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be02de81-3d2b-4368-94aa-13f9f33534ca-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:00:00.138048 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.138010    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll"]
Jul 17 01:00:00.138437 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.138419    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:00.138523 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.138439    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:00.138523 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.138470    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:00.138523 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.138479    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:00.138683 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.138572    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:00.138683 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.138586    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:00.141555 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.141535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll"
Jul 17 01:00:00.147685 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.147657    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll"]
Jul 17 01:00:00.153334 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.153273    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll"
Jul 17 01:00:00.281909 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.281744    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll"]
Jul 17 01:00:00.284649 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:00:00.284617    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbd9b93_b24f_4499_888c_d96665aed403.slice/crio-02b3101645da002df6fd2811da3b05775f55d87453108c6cd81ccaa27124d2fc WatchSource:0}: Error finding container 02b3101645da002df6fd2811da3b05775f55d87453108c6cd81ccaa27124d2fc: Status 404 returned error can't find the container with id 02b3101645da002df6fd2811da3b05775f55d87453108c6cd81ccaa27124d2fc
Jul 17 01:00:00.410977 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.410937    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll" event={"ID":"2bbd9b93-b24f-4499-888c-d96665aed403","Type":"ContainerStarted","Data":"02b3101645da002df6fd2811da3b05775f55d87453108c6cd81ccaa27124d2fc"}
Jul 17 01:00:00.412128 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.412109    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"
Jul 17 01:00:00.412267 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.412111    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz" event={"ID":"be02de81-3d2b-4368-94aa-13f9f33534ca","Type":"ContainerDied","Data":"301dcd1d0c1a8a87ada358a3e6e59f57eada41e08d0895ad3aabed1683db90c7"}
Jul 17 01:00:00.412267 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.412221    2573 scope.go:117] "RemoveContainer" containerID="6dfcfbff64051fe87966aeb3652a51c44712d897f6ec041ee3d6df309954ef62"
Jul 17 01:00:00.426457 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.426436    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"]
Jul 17 01:00:00.431647 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:00.431621    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-ls5bz"]
Jul 17 01:00:01.419410 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:01.419323    2573 generic.go:356] "Generic (PLEG): container finished" podID="2bbd9b93-b24f-4499-888c-d96665aed403" containerID="0814b1777a0e599faca634970f03c3c311507472d2679f9e1e2f332de9c27e94" exitCode=0
Jul 17 01:00:01.419854 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:01.419418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll" event={"ID":"2bbd9b93-b24f-4499-888c-d96665aed403","Type":"ContainerDied","Data":"0814b1777a0e599faca634970f03c3c311507472d2679f9e1e2f332de9c27e94"}
Jul 17 01:00:01.968353 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:01.968312    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" path="/var/lib/kubelet/pods/be02de81-3d2b-4368-94aa-13f9f33534ca/volumes"
Jul 17 01:00:02.558537 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:02.558511    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll"
Jul 17 01:00:02.962853 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:02.962822    2573 scope.go:117] "RemoveContainer" containerID="3ca2ea76a0d6ae8d620b4b119b5b3a4635c2ffeece9af7387299d47d588ceee3"
Jul 17 01:00:03.433607 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:03.433569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" event={"ID":"6db19fbb-3dda-4687-81f9-cb677a969b19","Type":"ContainerStarted","Data":"d785a1f33b33bab4f5baae14d8bac0a3f8bb12776e8a668810398d9464c4b923"}
Jul 17 01:00:03.434779 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:03.434756    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll"
Jul 17 01:00:03.434931 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:03.434811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737500-z8wll" event={"ID":"2bbd9b93-b24f-4499-888c-d96665aed403","Type":"ContainerDied","Data":"02b3101645da002df6fd2811da3b05775f55d87453108c6cd81ccaa27124d2fc"}
Jul 17 01:00:03.434931 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:03.434855    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02b3101645da002df6fd2811da3b05775f55d87453108c6cd81ccaa27124d2fc"
Jul 17 01:00:04.442197 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:04.442164    2573 generic.go:356] "Generic (PLEG): container finished" podID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerID="d785a1f33b33bab4f5baae14d8bac0a3f8bb12776e8a668810398d9464c4b923" exitCode=6
Jul 17 01:00:04.442618 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:04.442246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" event={"ID":"6db19fbb-3dda-4687-81f9-cb677a969b19","Type":"ContainerDied","Data":"d785a1f33b33bab4f5baae14d8bac0a3f8bb12776e8a668810398d9464c4b923"}
Jul 17 01:00:04.442618 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:04.442315    2573 scope.go:117] "RemoveContainer" containerID="3ca2ea76a0d6ae8d620b4b119b5b3a4635c2ffeece9af7387299d47d588ceee3"
Jul 17 01:00:04.442618 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:04.442532    2573 scope.go:117] "RemoveContainer" containerID="d785a1f33b33bab4f5baae14d8bac0a3f8bb12776e8a668810398d9464c4b923"
Jul 17 01:00:04.442769 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:04.442751    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-khgsz_odh-ai-gateway-infra(6db19fbb-3dda-4687-81f9-cb677a969b19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19"
Jul 17 01:00:04.457101 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:04.457077    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"]
Jul 17 01:00:05.587185 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:05.587159    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"
Jul 17 01:00:05.668449 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:05.668419    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19fbb-3dda-4687-81f9-cb677a969b19-maas-api-service-ca\") pod \"6db19fbb-3dda-4687-81f9-cb677a969b19\" (UID: \"6db19fbb-3dda-4687-81f9-cb677a969b19\") "
Jul 17 01:00:05.668775 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:05.668752    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db19fbb-3dda-4687-81f9-cb677a969b19-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6db19fbb-3dda-4687-81f9-cb677a969b19" (UID: "6db19fbb-3dda-4687-81f9-cb677a969b19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:00:05.769038 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:05.769004    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19fbb-3dda-4687-81f9-cb677a969b19-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:00:06.455665 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:06.455635    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"
Jul 17 01:00:06.455901 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:06.455630    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz" event={"ID":"6db19fbb-3dda-4687-81f9-cb677a969b19","Type":"ContainerDied","Data":"86940508488b71d7c31bab8ec25e624dfe87404f0fddb4b3e50e21002f1d5ed2"}
Jul 17 01:00:06.455901 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:06.455765    2573 scope.go:117] "RemoveContainer" containerID="d785a1f33b33bab4f5baae14d8bac0a3f8bb12776e8a668810398d9464c4b923"
Jul 17 01:00:06.471262 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:06.471231    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"]
Jul 17 01:00:06.480867 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:06.480846    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-khgsz"]
Jul 17 01:00:07.974039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:07.973960    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" path="/var/lib/kubelet/pods/6db19fbb-3dda-4687-81f9-cb677a969b19/volumes"
Jul 17 01:00:10.729311 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729278    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"]
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729620    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729630    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729648    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729653    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729665    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bbd9b93-b24f-4499-888c-d96665aed403" containerName="cleanup"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729671    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bbd9b93-b24f-4499-888c-d96665aed403" containerName="cleanup"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729679    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729685    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729693    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729699    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729748    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.729751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729755    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bbd9b93-b24f-4499-888c-d96665aed403" containerName="cleanup"
Jul 17 01:00:10.730210 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729762    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:10.730210 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.729768    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="be02de81-3d2b-4368-94aa-13f9f33534ca" containerName="revoke-keys"
Jul 17 01:00:10.733910 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.733883    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"
Jul 17 01:00:10.735977 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.735953    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:00:10.739591 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.739568    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"]
Jul 17 01:00:10.806689 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.806654    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2\" (UID: \"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"
Jul 17 01:00:10.907743 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.907717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2\" (UID: \"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"
Jul 17 01:00:10.908293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:10.908272    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2\" (UID: \"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"
Jul 17 01:00:11.044962 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:11.044930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"
Jul 17 01:00:11.173328 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:11.173295    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"]
Jul 17 01:00:11.174849 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:00:11.174774    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7fe046_53a4_4ae3_ac68_9bcdcb81daa9.slice/crio-6e4aea53cdd63629645ae4505ebaa2185a200d5a19e610e0c26486c2fb4d3655 WatchSource:0}: Error finding container 6e4aea53cdd63629645ae4505ebaa2185a200d5a19e610e0c26486c2fb4d3655: Status 404 returned error can't find the container with id 6e4aea53cdd63629645ae4505ebaa2185a200d5a19e610e0c26486c2fb4d3655
Jul 17 01:00:11.486200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:11.486167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" event={"ID":"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9","Type":"ContainerStarted","Data":"b0a0402f1643ccb318ee98cc789c96bb86f6897dcde945a948091a48eb6067bf"}
Jul 17 01:00:11.486356 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:11.486219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" event={"ID":"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9","Type":"ContainerStarted","Data":"6e4aea53cdd63629645ae4505ebaa2185a200d5a19e610e0c26486c2fb4d3655"}
Jul 17 01:00:11.501677 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:11.501628    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" podStartSLOduration=1.501615792 podStartE2EDuration="1.501615792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:00:11.499456093 +0000 UTC m=+2606.075608668" watchObservedRunningTime="2026-07-17 01:00:11.501615792 +0000 UTC m=+2606.077768411"
Jul 17 01:00:12.493331 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:12.493293    2573 generic.go:356] "Generic (PLEG): container finished" podID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerID="b0a0402f1643ccb318ee98cc789c96bb86f6897dcde945a948091a48eb6067bf" exitCode=6
Jul 17 01:00:12.493778 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:12.493377    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" event={"ID":"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9","Type":"ContainerDied","Data":"b0a0402f1643ccb318ee98cc789c96bb86f6897dcde945a948091a48eb6067bf"}
Jul 17 01:00:12.493778 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:12.493555    2573 scope.go:117] "RemoveContainer" containerID="b0a0402f1643ccb318ee98cc789c96bb86f6897dcde945a948091a48eb6067bf"
Jul 17 01:00:13.500829 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:13.500724    2573 generic.go:356] "Generic (PLEG): container finished" podID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerID="38773b4f89e6161d0c73f9e0e25b19bb49eba2edc6438104b68993c23bba97d1" exitCode=6
Jul 17 01:00:13.501218 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:13.500820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" event={"ID":"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9","Type":"ContainerDied","Data":"38773b4f89e6161d0c73f9e0e25b19bb49eba2edc6438104b68993c23bba97d1"}
Jul 17 01:00:13.501218 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:13.500880    2573 scope.go:117] "RemoveContainer" containerID="b0a0402f1643ccb318ee98cc789c96bb86f6897dcde945a948091a48eb6067bf"
Jul 17 01:00:13.501218 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:13.501043    2573 scope.go:117] "RemoveContainer" containerID="38773b4f89e6161d0c73f9e0e25b19bb49eba2edc6438104b68993c23bba97d1"
Jul 17 01:00:13.501350 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:13.501276    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2_odh-ai-gateway-infra(cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9"
Jul 17 01:00:14.509531 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:14.509507    2573 scope.go:117] "RemoveContainer" containerID="38773b4f89e6161d0c73f9e0e25b19bb49eba2edc6438104b68993c23bba97d1"
Jul 17 01:00:14.509949 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:14.509688    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2_odh-ai-gateway-infra(cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9"
Jul 17 01:00:23.171036 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.170998    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"]
Jul 17 01:00:23.171624 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.171597    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db19fbb-3dda-4687-81f9-cb677a969b19" containerName="revoke-keys"
Jul 17 01:00:23.174501 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.174484    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"
Jul 17 01:00:23.180573 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.180536    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"]
Jul 17 01:00:23.202757 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.202732    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9dc6f3-18df-438b-a94d-7a7d38b6999a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf\" (UID: \"8a9dc6f3-18df-438b-a94d-7a7d38b6999a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"
Jul 17 01:00:23.304027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.304004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9dc6f3-18df-438b-a94d-7a7d38b6999a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf\" (UID: \"8a9dc6f3-18df-438b-a94d-7a7d38b6999a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"
Jul 17 01:00:23.304574 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.304556    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9dc6f3-18df-438b-a94d-7a7d38b6999a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf\" (UID: \"8a9dc6f3-18df-438b-a94d-7a7d38b6999a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"
Jul 17 01:00:23.487358 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.487294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"
Jul 17 01:00:23.616136 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:23.616106    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"]
Jul 17 01:00:23.618483 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:00:23.618454    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9dc6f3_18df_438b_a94d_7a7d38b6999a.slice/crio-d97a2e283a01c45b101a1c700cdb551b06472d48d6e5830775f36b3942920464 WatchSource:0}: Error finding container d97a2e283a01c45b101a1c700cdb551b06472d48d6e5830775f36b3942920464: Status 404 returned error can't find the container with id d97a2e283a01c45b101a1c700cdb551b06472d48d6e5830775f36b3942920464
Jul 17 01:00:24.564552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:24.564516    2573 generic.go:356] "Generic (PLEG): container finished" podID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerID="03d21447f025715a0625d60afe9fcdc65e4d7d6a87c8e42beec12524c7965402" exitCode=6
Jul 17 01:00:24.564905 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:24.564597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" event={"ID":"8a9dc6f3-18df-438b-a94d-7a7d38b6999a","Type":"ContainerDied","Data":"03d21447f025715a0625d60afe9fcdc65e4d7d6a87c8e42beec12524c7965402"}
Jul 17 01:00:24.564905 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:24.564689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" event={"ID":"8a9dc6f3-18df-438b-a94d-7a7d38b6999a","Type":"ContainerStarted","Data":"d97a2e283a01c45b101a1c700cdb551b06472d48d6e5830775f36b3942920464"}
Jul 17 01:00:24.564905 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:24.564763    2573 scope.go:117] "RemoveContainer" containerID="03d21447f025715a0625d60afe9fcdc65e4d7d6a87c8e42beec12524c7965402"
Jul 17 01:00:25.572230 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:25.572199    2573 generic.go:356] "Generic (PLEG): container finished" podID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerID="671b4872944ba7854f0b4d85166956cace9e0eacd4c9b441da2bdab239014a6c" exitCode=6
Jul 17 01:00:25.572642 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:25.572252    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" event={"ID":"8a9dc6f3-18df-438b-a94d-7a7d38b6999a","Type":"ContainerDied","Data":"671b4872944ba7854f0b4d85166956cace9e0eacd4c9b441da2bdab239014a6c"}
Jul 17 01:00:25.572642 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:25.572314    2573 scope.go:117] "RemoveContainer" containerID="03d21447f025715a0625d60afe9fcdc65e4d7d6a87c8e42beec12524c7965402"
Jul 17 01:00:25.572642 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:25.572432    2573 scope.go:117] "RemoveContainer" containerID="671b4872944ba7854f0b4d85166956cace9e0eacd4c9b441da2bdab239014a6c"
Jul 17 01:00:25.572784 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:25.572636    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf_odh-ai-gateway-infra(8a9dc6f3-18df-438b-a94d-7a7d38b6999a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a"
Jul 17 01:00:25.964837 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:25.964760    2573 scope.go:117] "RemoveContainer" containerID="38773b4f89e6161d0c73f9e0e25b19bb49eba2edc6438104b68993c23bba97d1"
Jul 17 01:00:26.579615 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:26.579584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" event={"ID":"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9","Type":"ContainerStarted","Data":"cc8d2a5d040d5078de862d50adfae0e3bd82c72714c24628cc7a907fffd45b78"}
Jul 17 01:00:26.581291 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:26.581271    2573 scope.go:117] "RemoveContainer" containerID="671b4872944ba7854f0b4d85166956cace9e0eacd4c9b441da2bdab239014a6c"
Jul 17 01:00:26.581467 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:26.581449    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf_odh-ai-gateway-infra(8a9dc6f3-18df-438b-a94d-7a7d38b6999a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a"
Jul 17 01:00:27.588451 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:27.588416    2573 generic.go:356] "Generic (PLEG): container finished" podID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerID="cc8d2a5d040d5078de862d50adfae0e3bd82c72714c24628cc7a907fffd45b78" exitCode=6
Jul 17 01:00:27.588966 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:27.588493    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" event={"ID":"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9","Type":"ContainerDied","Data":"cc8d2a5d040d5078de862d50adfae0e3bd82c72714c24628cc7a907fffd45b78"}
Jul 17 01:00:27.588966 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:27.588582    2573 scope.go:117] "RemoveContainer" containerID="38773b4f89e6161d0c73f9e0e25b19bb49eba2edc6438104b68993c23bba97d1"
Jul 17 01:00:27.588966 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:27.588841    2573 scope.go:117] "RemoveContainer" containerID="cc8d2a5d040d5078de862d50adfae0e3bd82c72714c24628cc7a907fffd45b78"
Jul 17 01:00:27.589133 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:27.589079    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2_odh-ai-gateway-infra(cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9"
Jul 17 01:00:27.611836 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:27.611813    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"]
Jul 17 01:00:28.734847 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:28.734821    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"
Jul 17 01:00:28.845630 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:28.845598    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9-maas-api-service-ca\") pod \"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9\" (UID: \"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9\") "
Jul 17 01:00:28.845948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:28.845926    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" (UID: "cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:00:28.946385 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:28.946320    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:00:29.602985 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:29.602954    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"
Jul 17 01:00:29.603157 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:29.602983    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2" event={"ID":"cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9","Type":"ContainerDied","Data":"6e4aea53cdd63629645ae4505ebaa2185a200d5a19e610e0c26486c2fb4d3655"}
Jul 17 01:00:29.603157 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:29.603042    2573 scope.go:117] "RemoveContainer" containerID="cc8d2a5d040d5078de862d50adfae0e3bd82c72714c24628cc7a907fffd45b78"
Jul 17 01:00:29.624456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:29.624432    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"]
Jul 17 01:00:29.631145 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:29.631124    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-qpdc2"]
Jul 17 01:00:29.968096 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:29.968017    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" path="/var/lib/kubelet/pods/cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9/volumes"
Jul 17 01:00:37.962529 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:37.962497    2573 scope.go:117] "RemoveContainer" containerID="671b4872944ba7854f0b4d85166956cace9e0eacd4c9b441da2bdab239014a6c"
Jul 17 01:00:38.656109 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.656080    2573 generic.go:356] "Generic (PLEG): container finished" podID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerID="61b60ee2c6e3bd228142828cbbe8ff7bafb3e0ed2c9a967e7a585309471f24e3" exitCode=6
Jul 17 01:00:38.656240 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.656154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" event={"ID":"8a9dc6f3-18df-438b-a94d-7a7d38b6999a","Type":"ContainerDied","Data":"61b60ee2c6e3bd228142828cbbe8ff7bafb3e0ed2c9a967e7a585309471f24e3"}
Jul 17 01:00:38.656303 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.656242    2573 scope.go:117] "RemoveContainer" containerID="671b4872944ba7854f0b4d85166956cace9e0eacd4c9b441da2bdab239014a6c"
Jul 17 01:00:38.656513 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.656493    2573 scope.go:117] "RemoveContainer" containerID="61b60ee2c6e3bd228142828cbbe8ff7bafb3e0ed2c9a967e7a585309471f24e3"
Jul 17 01:00:38.656713 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:38.656693    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf_odh-ai-gateway-infra(8a9dc6f3-18df-438b-a94d-7a7d38b6999a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a"
Jul 17 01:00:38.910882 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.910800    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"]
Jul 17 01:00:38.911176 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911162    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.911245 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911177    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.911245 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911193    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.911245 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911198    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.911245 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911206    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.911245 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911213    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.911519 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911270    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.911519 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.911279    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:38.915511 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.915492    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"
Jul 17 01:00:38.920330 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:38.920308    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"]
Jul 17 01:00:39.026151 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.026118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5253fb-a5cc-46c0-8585-c6998d780ccd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w\" (UID: \"ae5253fb-a5cc-46c0-8585-c6998d780ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"
Jul 17 01:00:39.127311 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.127278    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5253fb-a5cc-46c0-8585-c6998d780ccd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w\" (UID: \"ae5253fb-a5cc-46c0-8585-c6998d780ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"
Jul 17 01:00:39.127858 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.127839    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5253fb-a5cc-46c0-8585-c6998d780ccd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w\" (UID: \"ae5253fb-a5cc-46c0-8585-c6998d780ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"
Jul 17 01:00:39.230051 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.229985    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"
Jul 17 01:00:39.364830 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.364783    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"]
Jul 17 01:00:39.366218 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:00:39.366190    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5253fb_a5cc_46c0_8585_c6998d780ccd.slice/crio-62e1494295efcd503d4de0f840ece5a8c2646723bf27fca682422f0c7ce964d9 WatchSource:0}: Error finding container 62e1494295efcd503d4de0f840ece5a8c2646723bf27fca682422f0c7ce964d9: Status 404 returned error can't find the container with id 62e1494295efcd503d4de0f840ece5a8c2646723bf27fca682422f0c7ce964d9
Jul 17 01:00:39.662440 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.662404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" event={"ID":"ae5253fb-a5cc-46c0-8585-c6998d780ccd","Type":"ContainerStarted","Data":"a6f493b985a69f2fb9cb701c91089aacd172928469ba6d194f85a9fecec09858"}
Jul 17 01:00:39.662557 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.662465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" event={"ID":"ae5253fb-a5cc-46c0-8585-c6998d780ccd","Type":"ContainerStarted","Data":"62e1494295efcd503d4de0f840ece5a8c2646723bf27fca682422f0c7ce964d9"}
Jul 17 01:00:39.674942 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.674716    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" podStartSLOduration=1.6746975210000001 podStartE2EDuration="1.674697521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:00:39.674301664 +0000 UTC m=+2634.250454255" watchObservedRunningTime="2026-07-17 01:00:39.674697521 +0000 UTC m=+2634.250850096"
Jul 17 01:00:39.679938 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.679915    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"]
Jul 17 01:00:39.810056 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.810034    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"
Jul 17 01:00:39.935343 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.935269    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9dc6f3-18df-438b-a94d-7a7d38b6999a-maas-api-service-ca\") pod \"8a9dc6f3-18df-438b-a94d-7a7d38b6999a\" (UID: \"8a9dc6f3-18df-438b-a94d-7a7d38b6999a\") "
Jul 17 01:00:39.935627 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:39.935607    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9dc6f3-18df-438b-a94d-7a7d38b6999a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a9dc6f3-18df-438b-a94d-7a7d38b6999a" (UID: "8a9dc6f3-18df-438b-a94d-7a7d38b6999a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:00:40.036834 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.036786    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9dc6f3-18df-438b-a94d-7a7d38b6999a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:00:40.670579 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.670546    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerID="a6f493b985a69f2fb9cb701c91089aacd172928469ba6d194f85a9fecec09858" exitCode=6
Jul 17 01:00:40.670764 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.670628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" event={"ID":"ae5253fb-a5cc-46c0-8585-c6998d780ccd","Type":"ContainerDied","Data":"a6f493b985a69f2fb9cb701c91089aacd172928469ba6d194f85a9fecec09858"}
Jul 17 01:00:40.670876 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.670861    2573 scope.go:117] "RemoveContainer" containerID="a6f493b985a69f2fb9cb701c91089aacd172928469ba6d194f85a9fecec09858"
Jul 17 01:00:40.672064 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.672040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf" event={"ID":"8a9dc6f3-18df-438b-a94d-7a7d38b6999a","Type":"ContainerDied","Data":"d97a2e283a01c45b101a1c700cdb551b06472d48d6e5830775f36b3942920464"}
Jul 17 01:00:40.672064 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.672063    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"
Jul 17 01:00:40.672229 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.672093    2573 scope.go:117] "RemoveContainer" containerID="61b60ee2c6e3bd228142828cbbe8ff7bafb3e0ed2c9a967e7a585309471f24e3"
Jul 17 01:00:40.701218 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.701194    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"]
Jul 17 01:00:40.704647 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:40.704627    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-rhwkf"]
Jul 17 01:00:41.679283 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:41.679206    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerID="0acea70638f59006cd61db26f51591a45335bd62837cafabbf503bb12eda13b6" exitCode=6
Jul 17 01:00:41.679671 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:41.679292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" event={"ID":"ae5253fb-a5cc-46c0-8585-c6998d780ccd","Type":"ContainerDied","Data":"0acea70638f59006cd61db26f51591a45335bd62837cafabbf503bb12eda13b6"}
Jul 17 01:00:41.679671 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:41.679346    2573 scope.go:117] "RemoveContainer" containerID="a6f493b985a69f2fb9cb701c91089aacd172928469ba6d194f85a9fecec09858"
Jul 17 01:00:41.679671 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:41.679515    2573 scope.go:117] "RemoveContainer" containerID="0acea70638f59006cd61db26f51591a45335bd62837cafabbf503bb12eda13b6"
Jul 17 01:00:41.679870 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:41.679708    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w_odh-ai-gateway-infra(ae5253fb-a5cc-46c0-8585-c6998d780ccd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd"
Jul 17 01:00:41.974142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:41.974069    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" path="/var/lib/kubelet/pods/8a9dc6f3-18df-438b-a94d-7a7d38b6999a/volumes"
Jul 17 01:00:42.687518 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:42.687492    2573 scope.go:117] "RemoveContainer" containerID="0acea70638f59006cd61db26f51591a45335bd62837cafabbf503bb12eda13b6"
Jul 17 01:00:42.687945 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:42.687670    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w_odh-ai-gateway-infra(ae5253fb-a5cc-46c0-8585-c6998d780ccd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd"
Jul 17 01:00:46.216846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.216806    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"]
Jul 17 01:00:46.217287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217184    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217196    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217206    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217211    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217219    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217225    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217289    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217650 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217300    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217650 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217306    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9dc6f3-18df-438b-a94d-7a7d38b6999a" containerName="revoke-keys"
Jul 17 01:00:46.217650 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.217313    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7fe046-53a4-4ae3-ac68-9bcdcb81daa9" containerName="revoke-keys"
Jul 17 01:00:46.221614 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.221596    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"
Jul 17 01:00:46.228256 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.228229    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"]
Jul 17 01:00:46.285701 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.285669    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0428e9f-aa37-4c4b-bd41-63d031b45912-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5bphk\" (UID: \"e0428e9f-aa37-4c4b-bd41-63d031b45912\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"
Jul 17 01:00:46.386214 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.386186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0428e9f-aa37-4c4b-bd41-63d031b45912-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5bphk\" (UID: \"e0428e9f-aa37-4c4b-bd41-63d031b45912\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"
Jul 17 01:00:46.386767 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.386747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0428e9f-aa37-4c4b-bd41-63d031b45912-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5bphk\" (UID: \"e0428e9f-aa37-4c4b-bd41-63d031b45912\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"
Jul 17 01:00:46.533380 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.533356    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"
Jul 17 01:00:46.672474 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.672442    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"]
Jul 17 01:00:46.675527 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:00:46.675491    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0428e9f_aa37_4c4b_bd41_63d031b45912.slice/crio-60b1fbe817c0c5b6a914ed34bfd892ac2756c3d689671ca4ce815b236d725960 WatchSource:0}: Error finding container 60b1fbe817c0c5b6a914ed34bfd892ac2756c3d689671ca4ce815b236d725960: Status 404 returned error can't find the container with id 60b1fbe817c0c5b6a914ed34bfd892ac2756c3d689671ca4ce815b236d725960
Jul 17 01:00:46.708953 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:46.708929    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" event={"ID":"e0428e9f-aa37-4c4b-bd41-63d031b45912","Type":"ContainerStarted","Data":"60b1fbe817c0c5b6a914ed34bfd892ac2756c3d689671ca4ce815b236d725960"}
Jul 17 01:00:47.716977 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:47.716897    2573 generic.go:356] "Generic (PLEG): container finished" podID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerID="264cbeb8c397885dc9d7d0ab17b2e58da907d5446cfa8728dac9de36ee16ee06" exitCode=6
Jul 17 01:00:47.717317 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:47.716974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" event={"ID":"e0428e9f-aa37-4c4b-bd41-63d031b45912","Type":"ContainerDied","Data":"264cbeb8c397885dc9d7d0ab17b2e58da907d5446cfa8728dac9de36ee16ee06"}
Jul 17 01:00:47.717317 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:47.717161    2573 scope.go:117] "RemoveContainer" containerID="264cbeb8c397885dc9d7d0ab17b2e58da907d5446cfa8728dac9de36ee16ee06"
Jul 17 01:00:48.724344 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:48.724261    2573 generic.go:356] "Generic (PLEG): container finished" podID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerID="22881a37c351ac912febb1c4d432873ffe8b2bfac16f5b1e7fb8cea09b059fee" exitCode=6
Jul 17 01:00:48.724344 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:48.724332    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" event={"ID":"e0428e9f-aa37-4c4b-bd41-63d031b45912","Type":"ContainerDied","Data":"22881a37c351ac912febb1c4d432873ffe8b2bfac16f5b1e7fb8cea09b059fee"}
Jul 17 01:00:48.724846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:48.724386    2573 scope.go:117] "RemoveContainer" containerID="264cbeb8c397885dc9d7d0ab17b2e58da907d5446cfa8728dac9de36ee16ee06"
Jul 17 01:00:48.724846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:48.724557    2573 scope.go:117] "RemoveContainer" containerID="22881a37c351ac912febb1c4d432873ffe8b2bfac16f5b1e7fb8cea09b059fee"
Jul 17 01:00:48.724846 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:48.724760    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5bphk_odh-ai-gateway-infra(e0428e9f-aa37-4c4b-bd41-63d031b45912)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912"
Jul 17 01:00:49.732114 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:49.732086    2573 scope.go:117] "RemoveContainer" containerID="22881a37c351ac912febb1c4d432873ffe8b2bfac16f5b1e7fb8cea09b059fee"
Jul 17 01:00:49.732484 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:49.732248    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5bphk_odh-ai-gateway-infra(e0428e9f-aa37-4c4b-bd41-63d031b45912)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912"
Jul 17 01:00:54.962539 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:54.962506    2573 scope.go:117] "RemoveContainer" containerID="0acea70638f59006cd61db26f51591a45335bd62837cafabbf503bb12eda13b6"
Jul 17 01:00:55.772320 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:55.772287    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerID="e4230b397ad5670a517251aa023e915aceddbc96678671feef9062da11e31e39" exitCode=6
Jul 17 01:00:55.772432 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:55.772355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" event={"ID":"ae5253fb-a5cc-46c0-8585-c6998d780ccd","Type":"ContainerDied","Data":"e4230b397ad5670a517251aa023e915aceddbc96678671feef9062da11e31e39"}
Jul 17 01:00:55.772432 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:55.772411    2573 scope.go:117] "RemoveContainer" containerID="0acea70638f59006cd61db26f51591a45335bd62837cafabbf503bb12eda13b6"
Jul 17 01:00:55.772733 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:55.772720    2573 scope.go:117] "RemoveContainer" containerID="e4230b397ad5670a517251aa023e915aceddbc96678671feef9062da11e31e39"
Jul 17 01:00:55.772941 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:00:55.772918    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w_odh-ai-gateway-infra(ae5253fb-a5cc-46c0-8585-c6998d780ccd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd"
Jul 17 01:00:56.796729 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:56.796697    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"]
Jul 17 01:00:56.936851 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:56.936824    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"
Jul 17 01:00:57.074659 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.074575    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5253fb-a5cc-46c0-8585-c6998d780ccd-maas-api-service-ca\") pod \"ae5253fb-a5cc-46c0-8585-c6998d780ccd\" (UID: \"ae5253fb-a5cc-46c0-8585-c6998d780ccd\") "
Jul 17 01:00:57.074957 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.074937    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5253fb-a5cc-46c0-8585-c6998d780ccd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae5253fb-a5cc-46c0-8585-c6998d780ccd" (UID: "ae5253fb-a5cc-46c0-8585-c6998d780ccd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:00:57.175441 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.175417    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5253fb-a5cc-46c0-8585-c6998d780ccd-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:00:57.785635 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.785607    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"
Jul 17 01:00:57.785837 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.785606    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w" event={"ID":"ae5253fb-a5cc-46c0-8585-c6998d780ccd","Type":"ContainerDied","Data":"62e1494295efcd503d4de0f840ece5a8c2646723bf27fca682422f0c7ce964d9"}
Jul 17 01:00:57.785837 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.785748    2573 scope.go:117] "RemoveContainer" containerID="e4230b397ad5670a517251aa023e915aceddbc96678671feef9062da11e31e39"
Jul 17 01:00:57.807781 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.807761    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"]
Jul 17 01:00:57.812644 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.812624    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-6sl5w"]
Jul 17 01:00:57.974708 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:00:57.974668    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" path="/var/lib/kubelet/pods/ae5253fb-a5cc-46c0-8585-c6998d780ccd/volumes"
Jul 17 01:01:00.769658 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.769628    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"]
Jul 17 01:01:00.770149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.770020    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:00.770149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.770032    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:00.770149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.770041    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:00.770149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.770048    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:00.770149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.770112    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:00.770149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.770118    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:00.774513 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.774497    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"
Jul 17 01:01:00.779866 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.779838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"]
Jul 17 01:01:00.907154 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:00.907130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc498ba-e5ce-4f08-9357-ef706eb4263d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8\" (UID: \"6dc498ba-e5ce-4f08-9357-ef706eb4263d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"
Jul 17 01:01:01.007975 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.007950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc498ba-e5ce-4f08-9357-ef706eb4263d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8\" (UID: \"6dc498ba-e5ce-4f08-9357-ef706eb4263d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"
Jul 17 01:01:01.008546 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.008527    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc498ba-e5ce-4f08-9357-ef706eb4263d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8\" (UID: \"6dc498ba-e5ce-4f08-9357-ef706eb4263d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"
Jul 17 01:01:01.085868 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.085818    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"
Jul 17 01:01:01.423518 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.423496    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"]
Jul 17 01:01:01.425333 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:01:01.425291    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc498ba_e5ce_4f08_9357_ef706eb4263d.slice/crio-58c1bd0d00d0ace78763f2097fb28a61c30d6ef7c4c5223c3fb4f06d86e0fb65 WatchSource:0}: Error finding container 58c1bd0d00d0ace78763f2097fb28a61c30d6ef7c4c5223c3fb4f06d86e0fb65: Status 404 returned error can't find the container with id 58c1bd0d00d0ace78763f2097fb28a61c30d6ef7c4c5223c3fb4f06d86e0fb65
Jul 17 01:01:01.809557 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.809531    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" event={"ID":"6dc498ba-e5ce-4f08-9357-ef706eb4263d","Type":"ContainerStarted","Data":"9aa3ec91776f0572440ea9d256d29eab27e5256474d6865e9410b6fe5b4e178b"}
Jul 17 01:01:01.809891 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.809580    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" event={"ID":"6dc498ba-e5ce-4f08-9357-ef706eb4263d","Type":"ContainerStarted","Data":"58c1bd0d00d0ace78763f2097fb28a61c30d6ef7c4c5223c3fb4f06d86e0fb65"}
Jul 17 01:01:01.825745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.825707    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" podStartSLOduration=1.82569134 podStartE2EDuration="1.82569134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:01:01.824906416 +0000 UTC m=+2656.401058992" watchObservedRunningTime="2026-07-17 01:01:01.82569134 +0000 UTC m=+2656.401843914"
Jul 17 01:01:01.962764 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:01.962745    2573 scope.go:117] "RemoveContainer" containerID="22881a37c351ac912febb1c4d432873ffe8b2bfac16f5b1e7fb8cea09b059fee"
Jul 17 01:01:02.816633 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:02.816598    2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerID="9aa3ec91776f0572440ea9d256d29eab27e5256474d6865e9410b6fe5b4e178b" exitCode=6
Jul 17 01:01:02.817058 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:02.816682    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" event={"ID":"6dc498ba-e5ce-4f08-9357-ef706eb4263d","Type":"ContainerDied","Data":"9aa3ec91776f0572440ea9d256d29eab27e5256474d6865e9410b6fe5b4e178b"}
Jul 17 01:01:02.817058 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:02.816886    2573 scope.go:117] "RemoveContainer" containerID="9aa3ec91776f0572440ea9d256d29eab27e5256474d6865e9410b6fe5b4e178b"
Jul 17 01:01:02.818560 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:02.818539    2573 generic.go:356] "Generic (PLEG): container finished" podID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerID="c758a51b709dd4aadd41d1fb46b8a06c00139bba40470955768f2da9cdf7fb24" exitCode=6
Jul 17 01:01:02.818686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:02.818606    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" event={"ID":"e0428e9f-aa37-4c4b-bd41-63d031b45912","Type":"ContainerDied","Data":"c758a51b709dd4aadd41d1fb46b8a06c00139bba40470955768f2da9cdf7fb24"}
Jul 17 01:01:02.818686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:02.818651    2573 scope.go:117] "RemoveContainer" containerID="22881a37c351ac912febb1c4d432873ffe8b2bfac16f5b1e7fb8cea09b059fee"
Jul 17 01:01:02.818912 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:02.818897    2573 scope.go:117] "RemoveContainer" containerID="c758a51b709dd4aadd41d1fb46b8a06c00139bba40470955768f2da9cdf7fb24"
Jul 17 01:01:02.819096 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:02.819076    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5bphk_odh-ai-gateway-infra(e0428e9f-aa37-4c4b-bd41-63d031b45912)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912"
Jul 17 01:01:03.827246 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:03.827161    2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerID="61c0302067f41ec3f2bd3c5e3c1a6d5adca9c7846f9525bc1a61b5215e8f1e5d" exitCode=6
Jul 17 01:01:03.827660 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:03.827245    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" event={"ID":"6dc498ba-e5ce-4f08-9357-ef706eb4263d","Type":"ContainerDied","Data":"61c0302067f41ec3f2bd3c5e3c1a6d5adca9c7846f9525bc1a61b5215e8f1e5d"}
Jul 17 01:01:03.827660 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:03.827300    2573 scope.go:117] "RemoveContainer" containerID="9aa3ec91776f0572440ea9d256d29eab27e5256474d6865e9410b6fe5b4e178b"
Jul 17 01:01:03.827660 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:03.827453    2573 scope.go:117] "RemoveContainer" containerID="61c0302067f41ec3f2bd3c5e3c1a6d5adca9c7846f9525bc1a61b5215e8f1e5d"
Jul 17 01:01:03.827816 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:03.827656    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8_odh-ai-gateway-infra(6dc498ba-e5ce-4f08-9357-ef706eb4263d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d"
Jul 17 01:01:03.851453 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:03.851428    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"]
Jul 17 01:01:03.996864 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:03.996840    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"
Jul 17 01:01:04.135200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.135135    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0428e9f-aa37-4c4b-bd41-63d031b45912-maas-api-service-ca\") pod \"e0428e9f-aa37-4c4b-bd41-63d031b45912\" (UID: \"e0428e9f-aa37-4c4b-bd41-63d031b45912\") "
Jul 17 01:01:04.135463 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.135442    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0428e9f-aa37-4c4b-bd41-63d031b45912-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0428e9f-aa37-4c4b-bd41-63d031b45912" (UID: "e0428e9f-aa37-4c4b-bd41-63d031b45912"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:01:04.236481 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.236457    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0428e9f-aa37-4c4b-bd41-63d031b45912-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:01:04.834086 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.834062    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"
Jul 17 01:01:04.834475 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.834098    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk" event={"ID":"e0428e9f-aa37-4c4b-bd41-63d031b45912","Type":"ContainerDied","Data":"60b1fbe817c0c5b6a914ed34bfd892ac2756c3d689671ca4ce815b236d725960"}
Jul 17 01:01:04.834475 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.834162    2573 scope.go:117] "RemoveContainer" containerID="c758a51b709dd4aadd41d1fb46b8a06c00139bba40470955768f2da9cdf7fb24"
Jul 17 01:01:04.836087 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.836068    2573 scope.go:117] "RemoveContainer" containerID="61c0302067f41ec3f2bd3c5e3c1a6d5adca9c7846f9525bc1a61b5215e8f1e5d"
Jul 17 01:01:04.836299 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:04.836276    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8_odh-ai-gateway-infra(6dc498ba-e5ce-4f08-9357-ef706eb4263d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d"
Jul 17 01:01:04.862686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.862667    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"]
Jul 17 01:01:04.867558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:04.867529    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5bphk"]
Jul 17 01:01:05.971085 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:05.971049    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" path="/var/lib/kubelet/pods/e0428e9f-aa37-4c4b-bd41-63d031b45912/volumes"
Jul 17 01:01:13.217242 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217212    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"]
Jul 17 01:01:13.217601 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217565    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:13.217601 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217577    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:13.217686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217602    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:13.217686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217609    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:13.217686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217628    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:13.217686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217634    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:13.217850 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217689    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5253fb-a5cc-46c0-8585-c6998d780ccd" containerName="revoke-keys"
Jul 17 01:01:13.217850 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217696    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:13.217850 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217703    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:13.217850 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.217709    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:13.221861 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.221843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"
Jul 17 01:01:13.226778 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.226746    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"]
Jul 17 01:01:13.307491 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.307452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af739a4d-6ca1-465c-99e9-511d0ae0cbb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks\" (UID: \"af739a4d-6ca1-465c-99e9-511d0ae0cbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"
Jul 17 01:01:13.407955 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.407920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af739a4d-6ca1-465c-99e9-511d0ae0cbb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks\" (UID: \"af739a4d-6ca1-465c-99e9-511d0ae0cbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"
Jul 17 01:01:13.408589 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.408572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af739a4d-6ca1-465c-99e9-511d0ae0cbb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks\" (UID: \"af739a4d-6ca1-465c-99e9-511d0ae0cbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"
Jul 17 01:01:13.534699 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.534672    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"
Jul 17 01:01:13.872834 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.872804    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"]
Jul 17 01:01:13.874131 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:01:13.874104    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf739a4d_6ca1_465c_99e9_511d0ae0cbb1.slice/crio-8421423fd1fd7ea27c686d18725ea363fff50cc0f9f3573e9cbc5dd6614f83d0 WatchSource:0}: Error finding container 8421423fd1fd7ea27c686d18725ea363fff50cc0f9f3573e9cbc5dd6614f83d0: Status 404 returned error can't find the container with id 8421423fd1fd7ea27c686d18725ea363fff50cc0f9f3573e9cbc5dd6614f83d0
Jul 17 01:01:13.886902 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:13.886873    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" event={"ID":"af739a4d-6ca1-465c-99e9-511d0ae0cbb1","Type":"ContainerStarted","Data":"8421423fd1fd7ea27c686d18725ea363fff50cc0f9f3573e9cbc5dd6614f83d0"}
Jul 17 01:01:14.893753 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:14.893716    2573 generic.go:356] "Generic (PLEG): container finished" podID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerID="0687b89685da1cfb61768c91a97fb0b18f26072bf33a976b26698e20edf317be" exitCode=6
Jul 17 01:01:14.894158 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:14.893785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" event={"ID":"af739a4d-6ca1-465c-99e9-511d0ae0cbb1","Type":"ContainerDied","Data":"0687b89685da1cfb61768c91a97fb0b18f26072bf33a976b26698e20edf317be"}
Jul 17 01:01:14.894158 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:14.894045    2573 scope.go:117] "RemoveContainer" containerID="0687b89685da1cfb61768c91a97fb0b18f26072bf33a976b26698e20edf317be"
Jul 17 01:01:15.901068 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:15.901031    2573 generic.go:356] "Generic (PLEG): container finished" podID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerID="7a12b702ca15cfa77544b46a7f068e8eb8aaeb192d26fbc2714120a3dba57674" exitCode=6
Jul 17 01:01:15.901456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:15.901126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" event={"ID":"af739a4d-6ca1-465c-99e9-511d0ae0cbb1","Type":"ContainerDied","Data":"7a12b702ca15cfa77544b46a7f068e8eb8aaeb192d26fbc2714120a3dba57674"}
Jul 17 01:01:15.901456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:15.901186    2573 scope.go:117] "RemoveContainer" containerID="0687b89685da1cfb61768c91a97fb0b18f26072bf33a976b26698e20edf317be"
Jul 17 01:01:15.901456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:15.901401    2573 scope.go:117] "RemoveContainer" containerID="7a12b702ca15cfa77544b46a7f068e8eb8aaeb192d26fbc2714120a3dba57674"
Jul 17 01:01:15.901626 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:15.901604    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks_odh-ai-gateway-infra(af739a4d-6ca1-465c-99e9-511d0ae0cbb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1"
Jul 17 01:01:16.908007 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:16.907985    2573 scope.go:117] "RemoveContainer" containerID="7a12b702ca15cfa77544b46a7f068e8eb8aaeb192d26fbc2714120a3dba57674"
Jul 17 01:01:16.908384 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:16.908140    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks_odh-ai-gateway-infra(af739a4d-6ca1-465c-99e9-511d0ae0cbb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1"
Jul 17 01:01:17.962413 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:17.962382    2573 scope.go:117] "RemoveContainer" containerID="61c0302067f41ec3f2bd3c5e3c1a6d5adca9c7846f9525bc1a61b5215e8f1e5d"
Jul 17 01:01:18.921560 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:18.921482    2573 generic.go:356] "Generic (PLEG): container finished" podID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerID="a7ac1d8c6ea55598081c0808df3284b4cb7d14d3c60d2943e2922545ced62702" exitCode=6
Jul 17 01:01:18.921706 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:18.921560    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" event={"ID":"6dc498ba-e5ce-4f08-9357-ef706eb4263d","Type":"ContainerDied","Data":"a7ac1d8c6ea55598081c0808df3284b4cb7d14d3c60d2943e2922545ced62702"}
Jul 17 01:01:18.921706 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:18.921626    2573 scope.go:117] "RemoveContainer" containerID="61c0302067f41ec3f2bd3c5e3c1a6d5adca9c7846f9525bc1a61b5215e8f1e5d"
Jul 17 01:01:18.921964 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:18.921944    2573 scope.go:117] "RemoveContainer" containerID="a7ac1d8c6ea55598081c0808df3284b4cb7d14d3c60d2943e2922545ced62702"
Jul 17 01:01:18.922146 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:18.922122    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8_odh-ai-gateway-infra(6dc498ba-e5ce-4f08-9357-ef706eb4263d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d"
Jul 17 01:01:19.945761 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:19.945728    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"]
Jul 17 01:01:20.078880 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.078854    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"
Jul 17 01:01:20.152785 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.152757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc498ba-e5ce-4f08-9357-ef706eb4263d-maas-api-service-ca\") pod \"6dc498ba-e5ce-4f08-9357-ef706eb4263d\" (UID: \"6dc498ba-e5ce-4f08-9357-ef706eb4263d\") "
Jul 17 01:01:20.153066 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.153046    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc498ba-e5ce-4f08-9357-ef706eb4263d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6dc498ba-e5ce-4f08-9357-ef706eb4263d" (UID: "6dc498ba-e5ce-4f08-9357-ef706eb4263d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:01:20.253477 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.253409    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc498ba-e5ce-4f08-9357-ef706eb4263d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:01:20.935660 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.935630    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"
Jul 17 01:01:20.935660 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.935640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8" event={"ID":"6dc498ba-e5ce-4f08-9357-ef706eb4263d","Type":"ContainerDied","Data":"58c1bd0d00d0ace78763f2097fb28a61c30d6ef7c4c5223c3fb4f06d86e0fb65"}
Jul 17 01:01:20.935927 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.935702    2573 scope.go:117] "RemoveContainer" containerID="a7ac1d8c6ea55598081c0808df3284b4cb7d14d3c60d2943e2922545ced62702"
Jul 17 01:01:20.956573 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.956548    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"]
Jul 17 01:01:20.960194 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:20.960171    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-fbwz8"]
Jul 17 01:01:21.968571 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:21.968541    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" path="/var/lib/kubelet/pods/6dc498ba-e5ce-4f08-9357-ef706eb4263d/volumes"
Jul 17 01:01:27.963004 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:27.962973    2573 scope.go:117] "RemoveContainer" containerID="7a12b702ca15cfa77544b46a7f068e8eb8aaeb192d26fbc2714120a3dba57674"
Jul 17 01:01:28.944513 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944444    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"]
Jul 17 01:01:28.944836 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944822    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.944836 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944836    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.944922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944847    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:28.944922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944853    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0428e9f-aa37-4c4b-bd41-63d031b45912" containerName="revoke-keys"
Jul 17 01:01:28.944922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944863    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.944922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944869    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.944922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.944922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944884    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.945098 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944938    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.945098 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944944    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.945098 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.944951    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc498ba-e5ce-4f08-9357-ef706eb4263d" containerName="revoke-keys"
Jul 17 01:01:28.947885 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.947868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"
Jul 17 01:01:28.954490 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.954456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"]
Jul 17 01:01:28.994993 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.994967    2573 generic.go:356] "Generic (PLEG): container finished" podID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerID="d488fecaa844377d24949fb527bd5618097c952f139ad8787312fd310f773563" exitCode=6
Jul 17 01:01:28.995321 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.995041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" event={"ID":"af739a4d-6ca1-465c-99e9-511d0ae0cbb1","Type":"ContainerDied","Data":"d488fecaa844377d24949fb527bd5618097c952f139ad8787312fd310f773563"}
Jul 17 01:01:28.995321 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.995113    2573 scope.go:117] "RemoveContainer" containerID="7a12b702ca15cfa77544b46a7f068e8eb8aaeb192d26fbc2714120a3dba57674"
Jul 17 01:01:28.995431 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:28.995393    2573 scope.go:117] "RemoveContainer" containerID="d488fecaa844377d24949fb527bd5618097c952f139ad8787312fd310f773563"
Jul 17 01:01:28.995587 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:28.995565    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks_odh-ai-gateway-infra(af739a4d-6ca1-465c-99e9-511d0ae0cbb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1"
Jul 17 01:01:29.013067 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:29.013039    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc\" (UID: \"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"
Jul 17 01:01:29.114304 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:29.114268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc\" (UID: \"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"
Jul 17 01:01:29.114781 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:29.114764    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc\" (UID: \"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"
Jul 17 01:01:29.259736 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:29.259712    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"
Jul 17 01:01:29.386629 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:29.386600    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"]
Jul 17 01:01:29.388293 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:01:29.388269    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f553ab_3d42_46b1_a1a4_b88ea5ed9c9b.slice/crio-6b5b11a16b1e91be57f47e827537bf77f4604fa7aeaf24ad9ca62218fd7566ad WatchSource:0}: Error finding container 6b5b11a16b1e91be57f47e827537bf77f4604fa7aeaf24ad9ca62218fd7566ad: Status 404 returned error can't find the container with id 6b5b11a16b1e91be57f47e827537bf77f4604fa7aeaf24ad9ca62218fd7566ad
Jul 17 01:01:30.002238 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.002131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" event={"ID":"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b","Type":"ContainerStarted","Data":"8f92fa3f586d4291563538908809dbf5909adda1f9ecfdbe6ef0e35a4209c131"}
Jul 17 01:01:30.002238 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.002193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" event={"ID":"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b","Type":"ContainerStarted","Data":"6b5b11a16b1e91be57f47e827537bf77f4604fa7aeaf24ad9ca62218fd7566ad"}
Jul 17 01:01:30.017161 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.017129    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"]
Jul 17 01:01:30.019316 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.019276    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" podStartSLOduration=2.019261623 podStartE2EDuration="2.019261623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:01:30.017326231 +0000 UTC m=+2684.593478830" watchObservedRunningTime="2026-07-17 01:01:30.019261623 +0000 UTC m=+2684.595414196"
Jul 17 01:01:30.147472 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.147450    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"
Jul 17 01:01:30.221667 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.221636    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af739a4d-6ca1-465c-99e9-511d0ae0cbb1-maas-api-service-ca\") pod \"af739a4d-6ca1-465c-99e9-511d0ae0cbb1\" (UID: \"af739a4d-6ca1-465c-99e9-511d0ae0cbb1\") "
Jul 17 01:01:30.222009 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.221988    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af739a4d-6ca1-465c-99e9-511d0ae0cbb1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af739a4d-6ca1-465c-99e9-511d0ae0cbb1" (UID: "af739a4d-6ca1-465c-99e9-511d0ae0cbb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:01:30.322811 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:30.322769    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af739a4d-6ca1-465c-99e9-511d0ae0cbb1-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:01:31.011232 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.011196    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerID="8f92fa3f586d4291563538908809dbf5909adda1f9ecfdbe6ef0e35a4209c131" exitCode=6
Jul 17 01:01:31.011662 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.011281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" event={"ID":"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b","Type":"ContainerDied","Data":"8f92fa3f586d4291563538908809dbf5909adda1f9ecfdbe6ef0e35a4209c131"}
Jul 17 01:01:31.011662 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.011410    2573 scope.go:117] "RemoveContainer" containerID="8f92fa3f586d4291563538908809dbf5909adda1f9ecfdbe6ef0e35a4209c131"
Jul 17 01:01:31.012649 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.012630    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"
Jul 17 01:01:31.012779 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.012632    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks" event={"ID":"af739a4d-6ca1-465c-99e9-511d0ae0cbb1","Type":"ContainerDied","Data":"8421423fd1fd7ea27c686d18725ea363fff50cc0f9f3573e9cbc5dd6614f83d0"}
Jul 17 01:01:31.012779 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.012771    2573 scope.go:117] "RemoveContainer" containerID="d488fecaa844377d24949fb527bd5618097c952f139ad8787312fd310f773563"
Jul 17 01:01:31.037725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.037703    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"]
Jul 17 01:01:31.040773 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.040752    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-c6hks"]
Jul 17 01:01:31.968364 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:31.968297    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" path="/var/lib/kubelet/pods/af739a4d-6ca1-465c-99e9-511d0ae0cbb1/volumes"
Jul 17 01:01:32.019384 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:32.019345    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerID="01234d90a4177564daf1f6930a694fe216ed034ad87f387f1e70604090f5b69f" exitCode=6
Jul 17 01:01:32.019816 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:32.019412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" event={"ID":"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b","Type":"ContainerDied","Data":"01234d90a4177564daf1f6930a694fe216ed034ad87f387f1e70604090f5b69f"}
Jul 17 01:01:32.019816 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:32.019469    2573 scope.go:117] "RemoveContainer" containerID="8f92fa3f586d4291563538908809dbf5909adda1f9ecfdbe6ef0e35a4209c131"
Jul 17 01:01:32.019816 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:32.019653    2573 scope.go:117] "RemoveContainer" containerID="01234d90a4177564daf1f6930a694fe216ed034ad87f387f1e70604090f5b69f"
Jul 17 01:01:32.019993 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:32.019866    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc_odh-ai-gateway-infra(c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b"
Jul 17 01:01:33.027418 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:33.027393    2573 scope.go:117] "RemoveContainer" containerID="01234d90a4177564daf1f6930a694fe216ed034ad87f387f1e70604090f5b69f"
Jul 17 01:01:33.027846 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:33.027564    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc_odh-ai-gateway-infra(c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b"
Jul 17 01:01:36.196387 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.196352    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"]
Jul 17 01:01:36.196971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.196953    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.197021 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.196976    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.197021 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.196989    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.197021 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.196998    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.197170 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.197096    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.197170 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.197127    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.197170 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.197138    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.202125 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.202105    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.204456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.204427    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 01:01:36.204687 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.204668    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-27f370-serving-cert\""
Jul 17 01:01:36.207636 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.207611    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"]
Jul 17 01:01:36.261221 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.261190    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"]
Jul 17 01:01:36.261643 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.261625    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.261643 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.261644    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="af739a4d-6ca1-465c-99e9-511d0ae0cbb1" containerName="revoke-keys"
Jul 17 01:01:36.264761 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.264745    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"
Jul 17 01:01:36.268395 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.268104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmw5\" (UniqueName: \"kubernetes.io/projected/566dec08-45d8-4b11-9f95-45e2341ed0a4-kube-api-access-dqmw5\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.268395 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.268156    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566dec08-45d8-4b11-9f95-45e2341ed0a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.268395 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.268233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.271857 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.271835    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"]
Jul 17 01:01:36.369120 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.369089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ab0d7f-059f-4ff6-a166-1a61a24886fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-8d94d\" (UID: \"a8ab0d7f-059f-4ff6-a166-1a61a24886fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"
Jul 17 01:01:36.369251 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.369130    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmw5\" (UniqueName: \"kubernetes.io/projected/566dec08-45d8-4b11-9f95-45e2341ed0a4-kube-api-access-dqmw5\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.369251 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.369152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566dec08-45d8-4b11-9f95-45e2341ed0a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.369328 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.369260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.369400 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:36.369386    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-serving-cert: secret "maas-api-e2e-inf-a-27f370-serving-cert" not found
Jul 17 01:01:36.369450 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:36.369441    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls podName:566dec08-45d8-4b11-9f95-45e2341ed0a4 nodeName:}" failed. No retries permitted until 2026-07-17 01:01:36.86942423 +0000 UTC m=+2691.445576802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls") pod "maas-api-e2e-inf-a-27f370-76546579b8-x92q8" (UID: "566dec08-45d8-4b11-9f95-45e2341ed0a4") : secret "maas-api-e2e-inf-a-27f370-serving-cert" not found
Jul 17 01:01:36.369694 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.369678    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566dec08-45d8-4b11-9f95-45e2341ed0a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.377409 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.377390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmw5\" (UniqueName: \"kubernetes.io/projected/566dec08-45d8-4b11-9f95-45e2341ed0a4-kube-api-access-dqmw5\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.469692 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.469634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ab0d7f-059f-4ff6-a166-1a61a24886fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-8d94d\" (UID: \"a8ab0d7f-059f-4ff6-a166-1a61a24886fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"
Jul 17 01:01:36.470213 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.470193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ab0d7f-059f-4ff6-a166-1a61a24886fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-8d94d\" (UID: \"a8ab0d7f-059f-4ff6-a166-1a61a24886fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"
Jul 17 01:01:36.576413 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.576393    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"
Jul 17 01:01:36.720948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.720923    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"]
Jul 17 01:01:36.722743 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:01:36.722710    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ab0d7f_059f_4ff6_a166_1a61a24886fd.slice/crio-03aebd779850ece7f425cecef3cfcc929d675cd416ae2db7c61c75342bfa5b3c WatchSource:0}: Error finding container 03aebd779850ece7f425cecef3cfcc929d675cd416ae2db7c61c75342bfa5b3c: Status 404 returned error can't find the container with id 03aebd779850ece7f425cecef3cfcc929d675cd416ae2db7c61c75342bfa5b3c
Jul 17 01:01:36.873537 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.873504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:36.875808 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:36.875770    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-27f370-76546579b8-x92q8\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:37.052715 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:37.052682    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" event={"ID":"a8ab0d7f-059f-4ff6-a166-1a61a24886fd","Type":"ContainerStarted","Data":"a79a9a2860e11e8629ed52c6e766407b41147bebb88389326341595d6ac6486c"}
Jul 17 01:01:37.052896 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:37.052736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" event={"ID":"a8ab0d7f-059f-4ff6-a166-1a61a24886fd","Type":"ContainerStarted","Data":"03aebd779850ece7f425cecef3cfcc929d675cd416ae2db7c61c75342bfa5b3c"}
Jul 17 01:01:37.068619 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:37.068570    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" podStartSLOduration=1.068555631 podStartE2EDuration="1.068555631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:01:37.06675986 +0000 UTC m=+2691.642912447" watchObservedRunningTime="2026-07-17 01:01:37.068555631 +0000 UTC m=+2691.644708206"
Jul 17 01:01:37.114639 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:37.114614    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:37.250213 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:37.250185    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"]
Jul 17 01:01:37.250943 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:01:37.250917    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566dec08_45d8_4b11_9f95_45e2341ed0a4.slice/crio-3f781c7e1b700516e232b84a0bc2d69203cd1d025f93f6b20de6ccdb65334aed WatchSource:0}: Error finding container 3f781c7e1b700516e232b84a0bc2d69203cd1d025f93f6b20de6ccdb65334aed: Status 404 returned error can't find the container with id 3f781c7e1b700516e232b84a0bc2d69203cd1d025f93f6b20de6ccdb65334aed
Jul 17 01:01:38.059417 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:38.059378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8" event={"ID":"566dec08-45d8-4b11-9f95-45e2341ed0a4","Type":"ContainerStarted","Data":"ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428"}
Jul 17 01:01:38.059616 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:38.059447    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8" event={"ID":"566dec08-45d8-4b11-9f95-45e2341ed0a4","Type":"ContainerStarted","Data":"3f781c7e1b700516e232b84a0bc2d69203cd1d025f93f6b20de6ccdb65334aed"}
Jul 17 01:01:38.059616 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:38.059480    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:38.060862 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:38.060839    2573 generic.go:356] "Generic (PLEG): container finished" podID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerID="a79a9a2860e11e8629ed52c6e766407b41147bebb88389326341595d6ac6486c" exitCode=6
Jul 17 01:01:38.060974 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:38.060889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" event={"ID":"a8ab0d7f-059f-4ff6-a166-1a61a24886fd","Type":"ContainerDied","Data":"a79a9a2860e11e8629ed52c6e766407b41147bebb88389326341595d6ac6486c"}
Jul 17 01:01:38.061072 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:38.061060    2573 scope.go:117] "RemoveContainer" containerID="a79a9a2860e11e8629ed52c6e766407b41147bebb88389326341595d6ac6486c"
Jul 17 01:01:38.076328 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:38.076288    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8" podStartSLOduration=1.6641297769999999 podStartE2EDuration="2.076274154s" totalImagesPullingTime="412.144377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:01:38.073802819 +0000 UTC m=+2692.649955390" watchObservedRunningTime="2026-07-17 01:01:38.076274154 +0000 UTC m=+2692.652427029"
Jul 17 01:01:39.068241 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:39.068207    2573 generic.go:356] "Generic (PLEG): container finished" podID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerID="956b6002cb735d6cde4161cd180f3ba0f54cfcce9c34ce5750b4727a49f54e5f" exitCode=6
Jul 17 01:01:39.068666 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:39.068292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" event={"ID":"a8ab0d7f-059f-4ff6-a166-1a61a24886fd","Type":"ContainerDied","Data":"956b6002cb735d6cde4161cd180f3ba0f54cfcce9c34ce5750b4727a49f54e5f"}
Jul 17 01:01:39.068666 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:39.068354    2573 scope.go:117] "RemoveContainer" containerID="a79a9a2860e11e8629ed52c6e766407b41147bebb88389326341595d6ac6486c"
Jul 17 01:01:39.068666 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:39.068453    2573 scope.go:117] "RemoveContainer" containerID="956b6002cb735d6cde4161cd180f3ba0f54cfcce9c34ce5750b4727a49f54e5f"
Jul 17 01:01:39.068879 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:39.068665    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-8d94d_odh-ai-gateway-infra(a8ab0d7f-059f-4ff6-a166-1a61a24886fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd"
Jul 17 01:01:40.075910 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:40.075884    2573 scope.go:117] "RemoveContainer" containerID="956b6002cb735d6cde4161cd180f3ba0f54cfcce9c34ce5750b4727a49f54e5f"
Jul 17 01:01:40.076261 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:40.076070    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-8d94d_odh-ai-gateway-infra(a8ab0d7f-059f-4ff6-a166-1a61a24886fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd"
Jul 17 01:01:44.074487 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.074460    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:01:44.090388 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.090331    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"]
Jul 17 01:01:44.099032 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.098993    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.104272 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.104242    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e1e207-serving-cert\""
Jul 17 01:01:44.108779 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.108751    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"]
Jul 17 01:01:44.132466 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.132444    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.132564 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.132483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.132621 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.132584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jmx\" (UniqueName: \"kubernetes.io/projected/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-kube-api-access-h8jmx\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.233448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.233415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.233594 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.233468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jmx\" (UniqueName: \"kubernetes.io/projected/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-kube-api-access-h8jmx\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.233594 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.233531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.234082 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.234054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.235948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.235923    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.243590 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.243564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jmx\" (UniqueName: \"kubernetes.io/projected/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-kube-api-access-h8jmx\") pod \"maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.416718 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.416648    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:44.557063 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.557037    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"]
Jul 17 01:01:44.559333 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:01:44.559303    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeae9d43_2a24_4a1d_aebe_875c780d6e1a.slice/crio-1e4f58054c029e4c3549d3a4e908235b1b59189f5183c11bb0186d57a805e322 WatchSource:0}: Error finding container 1e4f58054c029e4c3549d3a4e908235b1b59189f5183c11bb0186d57a805e322: Status 404 returned error can't find the container with id 1e4f58054c029e4c3549d3a4e908235b1b59189f5183c11bb0186d57a805e322
Jul 17 01:01:44.962623 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:44.962594    2573 scope.go:117] "RemoveContainer" containerID="01234d90a4177564daf1f6930a694fe216ed034ad87f387f1e70604090f5b69f"
Jul 17 01:01:45.116042 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:45.116010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl" event={"ID":"eeae9d43-2a24-4a1d-aebe-875c780d6e1a","Type":"ContainerStarted","Data":"b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f"}
Jul 17 01:01:45.116433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:45.116063    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl" event={"ID":"eeae9d43-2a24-4a1d-aebe-875c780d6e1a","Type":"ContainerStarted","Data":"1e4f58054c029e4c3549d3a4e908235b1b59189f5183c11bb0186d57a805e322"}
Jul 17 01:01:45.116433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:45.116117    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:45.178729 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:45.178684    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl" podStartSLOduration=0.747708274 podStartE2EDuration="1.178669385s" totalImagesPullingTime="430.961111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:01:45.176080368 +0000 UTC m=+2699.752232943" watchObservedRunningTime="2026-07-17 01:01:45.178669385 +0000 UTC m=+2699.754821958"
Jul 17 01:01:46.123381 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:46.123354    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerID="ec373aabb73f1175fcf10cac2211f28a840b98d97ead005c94ccefbef47854ac" exitCode=6
Jul 17 01:01:46.123744 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:46.123428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" event={"ID":"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b","Type":"ContainerDied","Data":"ec373aabb73f1175fcf10cac2211f28a840b98d97ead005c94ccefbef47854ac"}
Jul 17 01:01:46.123744 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:46.123487    2573 scope.go:117] "RemoveContainer" containerID="01234d90a4177564daf1f6930a694fe216ed034ad87f387f1e70604090f5b69f"
Jul 17 01:01:46.123884 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:46.123763    2573 scope.go:117] "RemoveContainer" containerID="ec373aabb73f1175fcf10cac2211f28a840b98d97ead005c94ccefbef47854ac"
Jul 17 01:01:46.124035 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:46.124009    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc_odh-ai-gateway-infra(c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b"
Jul 17 01:01:47.148447 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:47.148413    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"]
Jul 17 01:01:47.285004 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:47.284977    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"
Jul 17 01:01:47.359901 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:47.359875    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b-maas-api-service-ca\") pod \"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b\" (UID: \"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b\") "
Jul 17 01:01:47.360206 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:47.360184    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" (UID: "c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:01:47.460601 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:47.460543    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:01:47.907294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:47.907201    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 01:01:47.937348 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:47.917014    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 01:01:48.144175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:48.144151    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"
Jul 17 01:01:48.144324 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:48.144146    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc" event={"ID":"c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b","Type":"ContainerDied","Data":"6b5b11a16b1e91be57f47e827537bf77f4604fa7aeaf24ad9ca62218fd7566ad"}
Jul 17 01:01:48.144324 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:48.144289    2573 scope.go:117] "RemoveContainer" containerID="ec373aabb73f1175fcf10cac2211f28a840b98d97ead005c94ccefbef47854ac"
Jul 17 01:01:48.159558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:48.159504    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"]
Jul 17 01:01:48.166986 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:48.166965    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dlxzc"]
Jul 17 01:01:49.968646 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:49.968614    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" path="/var/lib/kubelet/pods/c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b/volumes"
Jul 17 01:01:51.129673 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:51.129644    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:01:51.962572 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:51.962542    2573 scope.go:117] "RemoveContainer" containerID="956b6002cb735d6cde4161cd180f3ba0f54cfcce9c34ce5750b4727a49f54e5f"
Jul 17 01:01:53.178854 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:53.178823    2573 generic.go:356] "Generic (PLEG): container finished" podID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerID="95bdee74f33ebc0d83dd55931e6e00ccc51905063f39ce322d4bfda027483f4f" exitCode=6
Jul 17 01:01:53.179252 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:53.178863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" event={"ID":"a8ab0d7f-059f-4ff6-a166-1a61a24886fd","Type":"ContainerDied","Data":"95bdee74f33ebc0d83dd55931e6e00ccc51905063f39ce322d4bfda027483f4f"}
Jul 17 01:01:53.179252 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:53.178911    2573 scope.go:117] "RemoveContainer" containerID="956b6002cb735d6cde4161cd180f3ba0f54cfcce9c34ce5750b4727a49f54e5f"
Jul 17 01:01:53.179252 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:53.179161    2573 scope.go:117] "RemoveContainer" containerID="95bdee74f33ebc0d83dd55931e6e00ccc51905063f39ce322d4bfda027483f4f"
Jul 17 01:01:53.179398 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:01:53.179349    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-8d94d_odh-ai-gateway-infra(a8ab0d7f-059f-4ff6-a166-1a61a24886fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd"
Jul 17 01:01:54.204299 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:54.204269    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"]
Jul 17 01:01:54.336310 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:54.336288    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"
Jul 17 01:01:54.421354 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:54.421326    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ab0d7f-059f-4ff6-a166-1a61a24886fd-maas-api-service-ca\") pod \"a8ab0d7f-059f-4ff6-a166-1a61a24886fd\" (UID: \"a8ab0d7f-059f-4ff6-a166-1a61a24886fd\") "
Jul 17 01:01:54.421655 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:54.421632    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ab0d7f-059f-4ff6-a166-1a61a24886fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8ab0d7f-059f-4ff6-a166-1a61a24886fd" (UID: "a8ab0d7f-059f-4ff6-a166-1a61a24886fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:01:54.522012 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:54.521987    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ab0d7f-059f-4ff6-a166-1a61a24886fd-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:01:55.193106 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:55.193076    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"
Jul 17 01:01:55.193287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:55.193105    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d" event={"ID":"a8ab0d7f-059f-4ff6-a166-1a61a24886fd","Type":"ContainerDied","Data":"03aebd779850ece7f425cecef3cfcc929d675cd416ae2db7c61c75342bfa5b3c"}
Jul 17 01:01:55.193287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:55.193168    2573 scope.go:117] "RemoveContainer" containerID="95bdee74f33ebc0d83dd55931e6e00ccc51905063f39ce322d4bfda027483f4f"
Jul 17 01:01:55.215347 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:55.215322    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"]
Jul 17 01:01:55.222483 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:55.222456    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-8d94d"]
Jul 17 01:01:55.968492 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:01:55.968460    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" path="/var/lib/kubelet/pods/a8ab0d7f-059f-4ff6-a166-1a61a24886fd/volumes"
Jul 17 01:02:00.813028 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.812989    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"]
Jul 17 01:02:00.813433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813387    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813399    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813409    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:00.813433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813419    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:00.813433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813431    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813437    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813445    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813451    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813464    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813469    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813528    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813536    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813542    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813547    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:00.813604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.813553    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f553ab-3d42-46b1-a1a4-b88ea5ed9c9b" containerName="revoke-keys"
Jul 17 01:02:00.818033 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.818016    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"
Jul 17 01:02:00.820031 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.820005    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:02:00.822488 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.822463    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"]
Jul 17 01:02:00.977075 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:00.977041    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39409750-9984-431b-ab0a-4a173201322c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt\" (UID: \"39409750-9984-431b-ab0a-4a173201322c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"
Jul 17 01:02:01.078175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:01.078096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39409750-9984-431b-ab0a-4a173201322c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt\" (UID: \"39409750-9984-431b-ab0a-4a173201322c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"
Jul 17 01:02:01.078760 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:01.078741    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39409750-9984-431b-ab0a-4a173201322c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt\" (UID: \"39409750-9984-431b-ab0a-4a173201322c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"
Jul 17 01:02:01.131029 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:01.130994    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"
Jul 17 01:02:01.268896 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:01.268872    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"]
Jul 17 01:02:01.270469 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:02:01.270442    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39409750_9984_431b_ab0a_4a173201322c.slice/crio-08875aa57c3f520e94a15d324ae0b14b0cdd0604e907bad066cc8011e02004db WatchSource:0}: Error finding container 08875aa57c3f520e94a15d324ae0b14b0cdd0604e907bad066cc8011e02004db: Status 404 returned error can't find the container with id 08875aa57c3f520e94a15d324ae0b14b0cdd0604e907bad066cc8011e02004db
Jul 17 01:02:02.241109 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:02.241026    2573 generic.go:356] "Generic (PLEG): container finished" podID="39409750-9984-431b-ab0a-4a173201322c" containerID="94172abfb684b6e619e50eda543f730d78aa86dc6feac28ac66ebe8c56ccfce3" exitCode=6
Jul 17 01:02:02.241506 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:02.241114    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" event={"ID":"39409750-9984-431b-ab0a-4a173201322c","Type":"ContainerDied","Data":"94172abfb684b6e619e50eda543f730d78aa86dc6feac28ac66ebe8c56ccfce3"}
Jul 17 01:02:02.241506 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:02.241177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" event={"ID":"39409750-9984-431b-ab0a-4a173201322c","Type":"ContainerStarted","Data":"08875aa57c3f520e94a15d324ae0b14b0cdd0604e907bad066cc8011e02004db"}
Jul 17 01:02:02.241506 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:02.241268    2573 scope.go:117] "RemoveContainer" containerID="94172abfb684b6e619e50eda543f730d78aa86dc6feac28ac66ebe8c56ccfce3"
Jul 17 01:02:03.248578 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.248487    2573 generic.go:356] "Generic (PLEG): container finished" podID="39409750-9984-431b-ab0a-4a173201322c" containerID="293aeb5d57b9b1fb76dddbb7a172696aa8de8b74973999fbdbfb76a1c5130146" exitCode=6
Jul 17 01:02:03.249077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.248575    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" event={"ID":"39409750-9984-431b-ab0a-4a173201322c","Type":"ContainerDied","Data":"293aeb5d57b9b1fb76dddbb7a172696aa8de8b74973999fbdbfb76a1c5130146"}
Jul 17 01:02:03.249077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.248663    2573 scope.go:117] "RemoveContainer" containerID="94172abfb684b6e619e50eda543f730d78aa86dc6feac28ac66ebe8c56ccfce3"
Jul 17 01:02:03.249077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.248867    2573 scope.go:117] "RemoveContainer" containerID="293aeb5d57b9b1fb76dddbb7a172696aa8de8b74973999fbdbfb76a1c5130146"
Jul 17 01:02:03.249077 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:03.249065    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt_odh-ai-gateway-infra(39409750-9984-431b-ab0a-4a173201322c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" podUID="39409750-9984-431b-ab0a-4a173201322c"
Jul 17 01:02:03.261509 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.261483    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"]
Jul 17 01:02:03.261991 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.261975    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:03.261991 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.261991    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:03.262112 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.262050    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ab0d7f-059f-4ff6-a166-1a61a24886fd" containerName="revoke-keys"
Jul 17 01:02:03.266339 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.266321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"
Jul 17 01:02:03.274337 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.274306    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"]
Jul 17 01:02:03.399403 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.399358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34c7e0d-8145-42fc-ad04-6b74a6fab7bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5\" (UID: \"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"
Jul 17 01:02:03.500780 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.500688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34c7e0d-8145-42fc-ad04-6b74a6fab7bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5\" (UID: \"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"
Jul 17 01:02:03.501324 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.501302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34c7e0d-8145-42fc-ad04-6b74a6fab7bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5\" (UID: \"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"
Jul 17 01:02:03.578262 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.578227    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"
Jul 17 01:02:03.707996 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:03.707970    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"]
Jul 17 01:02:03.708544 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:02:03.708522    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34c7e0d_8145_42fc_ad04_6b74a6fab7bd.slice/crio-2dd07a483ff08d0bb4242b903204faf12724286702227e2dcafd44008b9ffa64 WatchSource:0}: Error finding container 2dd07a483ff08d0bb4242b903204faf12724286702227e2dcafd44008b9ffa64: Status 404 returned error can't find the container with id 2dd07a483ff08d0bb4242b903204faf12724286702227e2dcafd44008b9ffa64
Jul 17 01:02:04.257162 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:04.257138    2573 scope.go:117] "RemoveContainer" containerID="293aeb5d57b9b1fb76dddbb7a172696aa8de8b74973999fbdbfb76a1c5130146"
Jul 17 01:02:04.257636 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:04.257368    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt_odh-ai-gateway-infra(39409750-9984-431b-ab0a-4a173201322c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" podUID="39409750-9984-431b-ab0a-4a173201322c"
Jul 17 01:02:04.258655 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:04.258628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" event={"ID":"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd","Type":"ContainerStarted","Data":"0f5e16e8834517f7c41902aa836100c6c95d310e7d3dfbcdcf818405405c3ef9"}
Jul 17 01:02:04.258776 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:04.258674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" event={"ID":"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd","Type":"ContainerStarted","Data":"2dd07a483ff08d0bb4242b903204faf12724286702227e2dcafd44008b9ffa64"}
Jul 17 01:02:04.283957 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:04.283900    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" podStartSLOduration=1.28388494 podStartE2EDuration="1.28388494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:02:04.281853861 +0000 UTC m=+2718.858006436" watchObservedRunningTime="2026-07-17 01:02:04.28388494 +0000 UTC m=+2718.860037513"
Jul 17 01:02:05.266512 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:05.266477    2573 generic.go:356] "Generic (PLEG): container finished" podID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerID="0f5e16e8834517f7c41902aa836100c6c95d310e7d3dfbcdcf818405405c3ef9" exitCode=6
Jul 17 01:02:05.266956 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:05.266528    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" event={"ID":"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd","Type":"ContainerDied","Data":"0f5e16e8834517f7c41902aa836100c6c95d310e7d3dfbcdcf818405405c3ef9"}
Jul 17 01:02:05.266956 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:05.266736    2573 scope.go:117] "RemoveContainer" containerID="0f5e16e8834517f7c41902aa836100c6c95d310e7d3dfbcdcf818405405c3ef9"
Jul 17 01:02:06.275963 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:06.275926    2573 generic.go:356] "Generic (PLEG): container finished" podID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerID="b67b17d13d8682eaf56d663f9c40a6e6d1ebd599b01beaf5942dd8a632c1892b" exitCode=6
Jul 17 01:02:06.276434 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:06.276015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" event={"ID":"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd","Type":"ContainerDied","Data":"b67b17d13d8682eaf56d663f9c40a6e6d1ebd599b01beaf5942dd8a632c1892b"}
Jul 17 01:02:06.276434 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:06.276098    2573 scope.go:117] "RemoveContainer" containerID="0f5e16e8834517f7c41902aa836100c6c95d310e7d3dfbcdcf818405405c3ef9"
Jul 17 01:02:06.276434 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:06.276357    2573 scope.go:117] "RemoveContainer" containerID="b67b17d13d8682eaf56d663f9c40a6e6d1ebd599b01beaf5942dd8a632c1892b"
Jul 17 01:02:06.276622 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:06.276563    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5_odh-ai-gateway-infra(c34c7e0d-8145-42fc-ad04-6b74a6fab7bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd"
Jul 17 01:02:07.286297 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:07.286268    2573 scope.go:117] "RemoveContainer" containerID="b67b17d13d8682eaf56d663f9c40a6e6d1ebd599b01beaf5942dd8a632c1892b"
Jul 17 01:02:07.286662 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:07.286473    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5_odh-ai-gateway-infra(c34c7e0d-8145-42fc-ad04-6b74a6fab7bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd"
Jul 17 01:02:17.962775 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:17.962740    2573 scope.go:117] "RemoveContainer" containerID="293aeb5d57b9b1fb76dddbb7a172696aa8de8b74973999fbdbfb76a1c5130146"
Jul 17 01:02:18.348136 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:18.348102    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" event={"ID":"39409750-9984-431b-ab0a-4a173201322c","Type":"ContainerStarted","Data":"faf98077d4c55327eeea5f11222778066c4359603146f768e9ae81a5fcba7fc7"}
Jul 17 01:02:18.364288 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:18.364241    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" podStartSLOduration=18.364226489 podStartE2EDuration="18.364226489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:02:18.361654009 +0000 UTC m=+2732.937806584" watchObservedRunningTime="2026-07-17 01:02:18.364226489 +0000 UTC m=+2732.940379063"
Jul 17 01:02:18.989069 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:18.988986    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"]
Jul 17 01:02:18.992416 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:18.992399    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"
Jul 17 01:02:18.998848 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:18.998770    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"]
Jul 17 01:02:19.128959 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.128920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab1b691-9433-4454-8f80-d55ca8c41e4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2\" (UID: \"9ab1b691-9433-4454-8f80-d55ca8c41e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"
Jul 17 01:02:19.230277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.230239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab1b691-9433-4454-8f80-d55ca8c41e4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2\" (UID: \"9ab1b691-9433-4454-8f80-d55ca8c41e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"
Jul 17 01:02:19.230889 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.230867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab1b691-9433-4454-8f80-d55ca8c41e4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2\" (UID: \"9ab1b691-9433-4454-8f80-d55ca8c41e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"
Jul 17 01:02:19.304920 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.304890    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"
Jul 17 01:02:19.356513 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.356486    2573 generic.go:356] "Generic (PLEG): container finished" podID="39409750-9984-431b-ab0a-4a173201322c" containerID="faf98077d4c55327eeea5f11222778066c4359603146f768e9ae81a5fcba7fc7" exitCode=6
Jul 17 01:02:19.356608 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.356550    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" event={"ID":"39409750-9984-431b-ab0a-4a173201322c","Type":"ContainerDied","Data":"faf98077d4c55327eeea5f11222778066c4359603146f768e9ae81a5fcba7fc7"}
Jul 17 01:02:19.356653 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.356639    2573 scope.go:117] "RemoveContainer" containerID="293aeb5d57b9b1fb76dddbb7a172696aa8de8b74973999fbdbfb76a1c5130146"
Jul 17 01:02:19.356980 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.356964    2573 scope.go:117] "RemoveContainer" containerID="faf98077d4c55327eeea5f11222778066c4359603146f768e9ae81a5fcba7fc7"
Jul 17 01:02:19.357267 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:19.357223    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt_odh-ai-gateway-infra(39409750-9984-431b-ab0a-4a173201322c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" podUID="39409750-9984-431b-ab0a-4a173201322c"
Jul 17 01:02:19.375145 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.375114    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"]
Jul 17 01:02:19.441566 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.441536    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"]
Jul 17 01:02:19.442783 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:02:19.442753    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab1b691_9433_4454_8f80_d55ca8c41e4d.slice/crio-4298e90caea521f5efacf38a56902a2e48e504b66b9d1ee44510c50754275d37 WatchSource:0}: Error finding container 4298e90caea521f5efacf38a56902a2e48e504b66b9d1ee44510c50754275d37: Status 404 returned error can't find the container with id 4298e90caea521f5efacf38a56902a2e48e504b66b9d1ee44510c50754275d37
Jul 17 01:02:19.962931 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:19.962848    2573 scope.go:117] "RemoveContainer" containerID="b67b17d13d8682eaf56d663f9c40a6e6d1ebd599b01beaf5942dd8a632c1892b"
Jul 17 01:02:20.363403 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.363370    2573 generic.go:356] "Generic (PLEG): container finished" podID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerID="4255185fc56ae7d36fd45463749b927c87aa3b05b180b4207fdbe08d2e784c1b" exitCode=6
Jul 17 01:02:20.363837 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.363447    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" event={"ID":"9ab1b691-9433-4454-8f80-d55ca8c41e4d","Type":"ContainerDied","Data":"4255185fc56ae7d36fd45463749b927c87aa3b05b180b4207fdbe08d2e784c1b"}
Jul 17 01:02:20.363837 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.363496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" event={"ID":"9ab1b691-9433-4454-8f80-d55ca8c41e4d","Type":"ContainerStarted","Data":"4298e90caea521f5efacf38a56902a2e48e504b66b9d1ee44510c50754275d37"}
Jul 17 01:02:20.363837 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.363631    2573 scope.go:117] "RemoveContainer" containerID="4255185fc56ae7d36fd45463749b927c87aa3b05b180b4207fdbe08d2e784c1b"
Jul 17 01:02:20.367205 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.367180    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" event={"ID":"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd","Type":"ContainerStarted","Data":"41a28887445d79c87da62b4cd54bd786422ba7b8ba1d806a57965dfae6ab4f81"}
Jul 17 01:02:20.507143 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.507118    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"
Jul 17 01:02:20.645368 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.645339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39409750-9984-431b-ab0a-4a173201322c-maas-api-service-ca\") pod \"39409750-9984-431b-ab0a-4a173201322c\" (UID: \"39409750-9984-431b-ab0a-4a173201322c\") "
Jul 17 01:02:20.645693 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.645667    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39409750-9984-431b-ab0a-4a173201322c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39409750-9984-431b-ab0a-4a173201322c" (UID: "39409750-9984-431b-ab0a-4a173201322c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:02:20.746975 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:20.746933    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39409750-9984-431b-ab0a-4a173201322c-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:02:21.374463 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.374431    2573 generic.go:356] "Generic (PLEG): container finished" podID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerID="8e395554005580f3917eac07894538223ee14a971b4b231adb0bc7aa3b2ae7b9" exitCode=6
Jul 17 01:02:21.374949 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.374516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" event={"ID":"9ab1b691-9433-4454-8f80-d55ca8c41e4d","Type":"ContainerDied","Data":"8e395554005580f3917eac07894538223ee14a971b4b231adb0bc7aa3b2ae7b9"}
Jul 17 01:02:21.374949 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.374580    2573 scope.go:117] "RemoveContainer" containerID="4255185fc56ae7d36fd45463749b927c87aa3b05b180b4207fdbe08d2e784c1b"
Jul 17 01:02:21.374949 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.374784    2573 scope.go:117] "RemoveContainer" containerID="8e395554005580f3917eac07894538223ee14a971b4b231adb0bc7aa3b2ae7b9"
Jul 17 01:02:21.375105 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:21.375070    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2_odh-ai-gateway-infra(9ab1b691-9433-4454-8f80-d55ca8c41e4d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d"
Jul 17 01:02:21.376252 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.376135    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"
Jul 17 01:02:21.376252 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.376155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt" event={"ID":"39409750-9984-431b-ab0a-4a173201322c","Type":"ContainerDied","Data":"08875aa57c3f520e94a15d324ae0b14b0cdd0604e907bad066cc8011e02004db"}
Jul 17 01:02:21.378187 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.378166    2573 generic.go:356] "Generic (PLEG): container finished" podID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerID="41a28887445d79c87da62b4cd54bd786422ba7b8ba1d806a57965dfae6ab4f81" exitCode=6
Jul 17 01:02:21.378276 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.378241    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" event={"ID":"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd","Type":"ContainerDied","Data":"41a28887445d79c87da62b4cd54bd786422ba7b8ba1d806a57965dfae6ab4f81"}
Jul 17 01:02:21.378573 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.378558    2573 scope.go:117] "RemoveContainer" containerID="41a28887445d79c87da62b4cd54bd786422ba7b8ba1d806a57965dfae6ab4f81"
Jul 17 01:02:21.378815 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:21.378777    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5_odh-ai-gateway-infra(c34c7e0d-8145-42fc-ad04-6b74a6fab7bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd"
Jul 17 01:02:21.387441 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.387421    2573 scope.go:117] "RemoveContainer" containerID="faf98077d4c55327eeea5f11222778066c4359603146f768e9ae81a5fcba7fc7"
Jul 17 01:02:21.396574 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.396546    2573 scope.go:117] "RemoveContainer" containerID="b67b17d13d8682eaf56d663f9c40a6e6d1ebd599b01beaf5942dd8a632c1892b"
Jul 17 01:02:21.401971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.401915    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"]
Jul 17 01:02:21.406221 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.406195    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-98nrt"]
Jul 17 01:02:21.412725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.412702    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"]
Jul 17 01:02:21.968561 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:21.968527    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39409750-9984-431b-ab0a-4a173201322c" path="/var/lib/kubelet/pods/39409750-9984-431b-ab0a-4a173201322c/volumes"
Jul 17 01:02:22.386056 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:22.386029    2573 scope.go:117] "RemoveContainer" containerID="8e395554005580f3917eac07894538223ee14a971b4b231adb0bc7aa3b2ae7b9"
Jul 17 01:02:22.386512 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:22.386237    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2_odh-ai-gateway-infra(9ab1b691-9433-4454-8f80-d55ca8c41e4d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d"
Jul 17 01:02:22.521981 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:22.521956    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"
Jul 17 01:02:22.562389 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:22.562358    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34c7e0d-8145-42fc-ad04-6b74a6fab7bd-maas-api-service-ca\") pod \"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd\" (UID: \"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd\") "
Jul 17 01:02:22.562710 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:22.562688    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34c7e0d-8145-42fc-ad04-6b74a6fab7bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" (UID: "c34c7e0d-8145-42fc-ad04-6b74a6fab7bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:02:22.663065 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:22.662960    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34c7e0d-8145-42fc-ad04-6b74a6fab7bd-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:02:23.395508 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:23.395476    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"
Jul 17 01:02:23.395949 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:23.395516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5" event={"ID":"c34c7e0d-8145-42fc-ad04-6b74a6fab7bd","Type":"ContainerDied","Data":"2dd07a483ff08d0bb4242b903204faf12724286702227e2dcafd44008b9ffa64"}
Jul 17 01:02:23.395949 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:23.395598    2573 scope.go:117] "RemoveContainer" containerID="41a28887445d79c87da62b4cd54bd786422ba7b8ba1d806a57965dfae6ab4f81"
Jul 17 01:02:23.415457 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:23.415427    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"]
Jul 17 01:02:23.418823 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:23.418801    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9k8k5"]
Jul 17 01:02:23.967887 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:23.967851    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" path="/var/lib/kubelet/pods/c34c7e0d-8145-42fc-ad04-6b74a6fab7bd/volumes"
Jul 17 01:02:26.306493 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306454    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"]
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306831    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306844    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306855    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306861    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306884    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306892    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.306897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306898    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306907    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306913    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306966    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306973    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306979    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306987    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306993    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:02:26.307152 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.306999    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34c7e0d-8145-42fc-ad04-6b74a6fab7bd" containerName="revoke-keys"
Jul 17 01:02:26.309974 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.309957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"
Jul 17 01:02:26.316188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.316156    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"]
Jul 17 01:02:26.397131 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.397103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14003462-eae6-442e-ab4c-c65fb5898b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-d7hpz\" (UID: \"14003462-eae6-442e-ab4c-c65fb5898b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"
Jul 17 01:02:26.498200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.498167    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14003462-eae6-442e-ab4c-c65fb5898b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-d7hpz\" (UID: \"14003462-eae6-442e-ab4c-c65fb5898b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"
Jul 17 01:02:26.498730 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.498713    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14003462-eae6-442e-ab4c-c65fb5898b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-d7hpz\" (UID: \"14003462-eae6-442e-ab4c-c65fb5898b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"
Jul 17 01:02:26.621784 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.621706    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"
Jul 17 01:02:26.964686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:26.964654    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"]
Jul 17 01:02:26.966419 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:02:26.966308    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14003462_eae6_442e_ab4c_c65fb5898b38.slice/crio-8f0b46c058089f66bc3748e89d51b073bce40b0c627b1aefcdd0a11d717f0c6d WatchSource:0}: Error finding container 8f0b46c058089f66bc3748e89d51b073bce40b0c627b1aefcdd0a11d717f0c6d: Status 404 returned error can't find the container with id 8f0b46c058089f66bc3748e89d51b073bce40b0c627b1aefcdd0a11d717f0c6d
Jul 17 01:02:27.418974 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:27.418942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" event={"ID":"14003462-eae6-442e-ab4c-c65fb5898b38","Type":"ContainerStarted","Data":"8f7b98dccb84fab115e1ada63c38219448718d63aebbd856864ce831849d1f0f"}
Jul 17 01:02:27.419272 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:27.419000    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" event={"ID":"14003462-eae6-442e-ab4c-c65fb5898b38","Type":"ContainerStarted","Data":"8f0b46c058089f66bc3748e89d51b073bce40b0c627b1aefcdd0a11d717f0c6d"}
Jul 17 01:02:27.433954 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:27.433908    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" podStartSLOduration=1.433895416 podStartE2EDuration="1.433895416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:02:27.431350846 +0000 UTC m=+2742.007503420" watchObservedRunningTime="2026-07-17 01:02:27.433895416 +0000 UTC m=+2742.010047990"
Jul 17 01:02:28.425649 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:28.425618    2573 generic.go:356] "Generic (PLEG): container finished" podID="14003462-eae6-442e-ab4c-c65fb5898b38" containerID="8f7b98dccb84fab115e1ada63c38219448718d63aebbd856864ce831849d1f0f" exitCode=6
Jul 17 01:02:28.426023 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:28.425664    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" event={"ID":"14003462-eae6-442e-ab4c-c65fb5898b38","Type":"ContainerDied","Data":"8f7b98dccb84fab115e1ada63c38219448718d63aebbd856864ce831849d1f0f"}
Jul 17 01:02:28.426023 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:28.425835    2573 scope.go:117] "RemoveContainer" containerID="8f7b98dccb84fab115e1ada63c38219448718d63aebbd856864ce831849d1f0f"
Jul 17 01:02:29.433719 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:29.433629    2573 generic.go:356] "Generic (PLEG): container finished" podID="14003462-eae6-442e-ab4c-c65fb5898b38" containerID="c0d5b2226d99bcef28d345397cbca33a21e8203e9d095f71c5c6019206ea2de9" exitCode=6
Jul 17 01:02:29.434165 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:29.433715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" event={"ID":"14003462-eae6-442e-ab4c-c65fb5898b38","Type":"ContainerDied","Data":"c0d5b2226d99bcef28d345397cbca33a21e8203e9d095f71c5c6019206ea2de9"}
Jul 17 01:02:29.434165 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:29.433849    2573 scope.go:117] "RemoveContainer" containerID="8f7b98dccb84fab115e1ada63c38219448718d63aebbd856864ce831849d1f0f"
Jul 17 01:02:29.434165 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:29.434026    2573 scope.go:117] "RemoveContainer" containerID="c0d5b2226d99bcef28d345397cbca33a21e8203e9d095f71c5c6019206ea2de9"
Jul 17 01:02:29.434308 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:29.434262    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-d7hpz_odh-ai-gateway-infra(14003462-eae6-442e-ab4c-c65fb5898b38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" podUID="14003462-eae6-442e-ab4c-c65fb5898b38"
Jul 17 01:02:30.442987 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:30.442955    2573 scope.go:117] "RemoveContainer" containerID="c0d5b2226d99bcef28d345397cbca33a21e8203e9d095f71c5c6019206ea2de9"
Jul 17 01:02:30.443426 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:30.443147    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-d7hpz_odh-ai-gateway-infra(14003462-eae6-442e-ab4c-c65fb5898b38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" podUID="14003462-eae6-442e-ab4c-c65fb5898b38"
Jul 17 01:02:34.962860 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:34.962816    2573 scope.go:117] "RemoveContainer" containerID="8e395554005580f3917eac07894538223ee14a971b4b231adb0bc7aa3b2ae7b9"
Jul 17 01:02:35.474041 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:35.474004    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" event={"ID":"9ab1b691-9433-4454-8f80-d55ca8c41e4d","Type":"ContainerStarted","Data":"37c1fa5d67943fae1ad9907c63f2c5c60fb1334b5994bb5491d7a4e94c516015"}
Jul 17 01:02:35.490980 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:35.490912    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" podStartSLOduration=17.490891191 podStartE2EDuration="17.490891191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:02:35.487910646 +0000 UTC m=+2750.064063221" watchObservedRunningTime="2026-07-17 01:02:35.490891191 +0000 UTC m=+2750.067043784"
Jul 17 01:02:36.482685 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:36.482633    2573 generic.go:356] "Generic (PLEG): container finished" podID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerID="37c1fa5d67943fae1ad9907c63f2c5c60fb1334b5994bb5491d7a4e94c516015" exitCode=6
Jul 17 01:02:36.483298 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:36.482712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" event={"ID":"9ab1b691-9433-4454-8f80-d55ca8c41e4d","Type":"ContainerDied","Data":"37c1fa5d67943fae1ad9907c63f2c5c60fb1334b5994bb5491d7a4e94c516015"}
Jul 17 01:02:36.483298 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:36.482831    2573 scope.go:117] "RemoveContainer" containerID="8e395554005580f3917eac07894538223ee14a971b4b231adb0bc7aa3b2ae7b9"
Jul 17 01:02:36.483298 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:36.483195    2573 scope.go:117] "RemoveContainer" containerID="37c1fa5d67943fae1ad9907c63f2c5c60fb1334b5994bb5491d7a4e94c516015"
Jul 17 01:02:36.483505 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:36.483478    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2_odh-ai-gateway-infra(9ab1b691-9433-4454-8f80-d55ca8c41e4d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d"
Jul 17 01:02:36.499500 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:36.499457    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"]
Jul 17 01:02:37.650157 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:37.650114    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"
Jul 17 01:02:37.705758 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:37.705704    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab1b691-9433-4454-8f80-d55ca8c41e4d-maas-api-service-ca\") pod \"9ab1b691-9433-4454-8f80-d55ca8c41e4d\" (UID: \"9ab1b691-9433-4454-8f80-d55ca8c41e4d\") "
Jul 17 01:02:37.706228 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:37.706198    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab1b691-9433-4454-8f80-d55ca8c41e4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ab1b691-9433-4454-8f80-d55ca8c41e4d" (UID: "9ab1b691-9433-4454-8f80-d55ca8c41e4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:02:37.807478 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:37.807436    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab1b691-9433-4454-8f80-d55ca8c41e4d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:02:38.499372 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:38.499332    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"
Jul 17 01:02:38.499372 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:38.499352    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2" event={"ID":"9ab1b691-9433-4454-8f80-d55ca8c41e4d","Type":"ContainerDied","Data":"4298e90caea521f5efacf38a56902a2e48e504b66b9d1ee44510c50754275d37"}
Jul 17 01:02:38.499716 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:38.499447    2573 scope.go:117] "RemoveContainer" containerID="37c1fa5d67943fae1ad9907c63f2c5c60fb1334b5994bb5491d7a4e94c516015"
Jul 17 01:02:38.517979 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:38.517932    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"]
Jul 17 01:02:38.523338 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:38.523290    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-dhxb2"]
Jul 17 01:02:39.968812 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:39.968751    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" path="/var/lib/kubelet/pods/9ab1b691-9433-4454-8f80-d55ca8c41e4d/volumes"
Jul 17 01:02:40.962420 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:40.962383    2573 scope.go:117] "RemoveContainer" containerID="c0d5b2226d99bcef28d345397cbca33a21e8203e9d095f71c5c6019206ea2de9"
Jul 17 01:02:41.523584 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:41.523538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" event={"ID":"14003462-eae6-442e-ab4c-c65fb5898b38","Type":"ContainerStarted","Data":"2ede08194509dddd8a2ca842d15c97da50cc0faa0664131a20b0280d3bc109db"}
Jul 17 01:02:42.531708 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:42.531671    2573 generic.go:356] "Generic (PLEG): container finished" podID="14003462-eae6-442e-ab4c-c65fb5898b38" containerID="2ede08194509dddd8a2ca842d15c97da50cc0faa0664131a20b0280d3bc109db" exitCode=6
Jul 17 01:02:42.532229 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:42.531749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" event={"ID":"14003462-eae6-442e-ab4c-c65fb5898b38","Type":"ContainerDied","Data":"2ede08194509dddd8a2ca842d15c97da50cc0faa0664131a20b0280d3bc109db"}
Jul 17 01:02:42.532229 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:42.531840    2573 scope.go:117] "RemoveContainer" containerID="c0d5b2226d99bcef28d345397cbca33a21e8203e9d095f71c5c6019206ea2de9"
Jul 17 01:02:42.532229 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:42.532117    2573 scope.go:117] "RemoveContainer" containerID="2ede08194509dddd8a2ca842d15c97da50cc0faa0664131a20b0280d3bc109db"
Jul 17 01:02:42.532349 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:02:42.532327    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-d7hpz_odh-ai-gateway-infra(14003462-eae6-442e-ab4c-c65fb5898b38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" podUID="14003462-eae6-442e-ab4c-c65fb5898b38"
Jul 17 01:02:42.557009 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:42.556966    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"]
Jul 17 01:02:43.699505 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:43.699470    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"
Jul 17 01:02:43.765003 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:43.764949    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14003462-eae6-442e-ab4c-c65fb5898b38-maas-api-service-ca\") pod \"14003462-eae6-442e-ab4c-c65fb5898b38\" (UID: \"14003462-eae6-442e-ab4c-c65fb5898b38\") "
Jul 17 01:02:43.765393 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:43.765366    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14003462-eae6-442e-ab4c-c65fb5898b38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14003462-eae6-442e-ab4c-c65fb5898b38" (UID: "14003462-eae6-442e-ab4c-c65fb5898b38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:02:43.866249 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:43.866135    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14003462-eae6-442e-ab4c-c65fb5898b38-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:02:44.548801 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:44.548751    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"
Jul 17 01:02:44.549006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:44.548784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz" event={"ID":"14003462-eae6-442e-ab4c-c65fb5898b38","Type":"ContainerDied","Data":"8f0b46c058089f66bc3748e89d51b073bce40b0c627b1aefcdd0a11d717f0c6d"}
Jul 17 01:02:44.549006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:44.548880    2573 scope.go:117] "RemoveContainer" containerID="2ede08194509dddd8a2ca842d15c97da50cc0faa0664131a20b0280d3bc109db"
Jul 17 01:02:44.567528 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:44.567474    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"]
Jul 17 01:02:44.570520 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:44.570466    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d7hpz"]
Jul 17 01:02:45.969215 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:45.969173    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" path="/var/lib/kubelet/pods/14003462-eae6-442e-ab4c-c65fb5898b38/volumes"
Jul 17 01:02:56.477741 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:02:56.477703    2573 scope.go:117] "RemoveContainer" containerID="348ce1dece7a144e0243dbc0b52f40b5ec2ab08b8fc126af11ede931d4cde984"
Jul 17 01:03:00.865160 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865116    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"]
Jul 17 01:03:00.865604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865569    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:00.865604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865582    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:00.865604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865595    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865604 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865602    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865609    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865615    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39409750-9984-431b-ab0a-4a173201322c" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865622    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865627    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865637    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865642    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865649    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865653    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865715    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865721    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865727    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865734    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865741    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab1b691-9433-4454-8f80-d55ca8c41e4d" containerName="revoke-keys"
Jul 17 01:03:00.865745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.865746    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:00.871084 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.871035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"
Jul 17 01:03:00.873261 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.873228    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:03:00.875371 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:00.875336    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"]
Jul 17 01:03:01.040190 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:01.040139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c857f5c-c90e-4232-a24a-1c377c0062c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp\" (UID: \"3c857f5c-c90e-4232-a24a-1c377c0062c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"
Jul 17 01:03:01.141259 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:01.141156    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c857f5c-c90e-4232-a24a-1c377c0062c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp\" (UID: \"3c857f5c-c90e-4232-a24a-1c377c0062c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"
Jul 17 01:03:01.141823 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:01.141780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c857f5c-c90e-4232-a24a-1c377c0062c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp\" (UID: \"3c857f5c-c90e-4232-a24a-1c377c0062c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"
Jul 17 01:03:01.185674 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:01.185628    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"
Jul 17 01:03:01.344045 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:01.344008    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"]
Jul 17 01:03:01.346680 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:03:01.346641    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c857f5c_c90e_4232_a24a_1c377c0062c7.slice/crio-0b8c2ace276c4224edde4a734104fffc51b41e44360bf9a2624e86cb56d6ccc5 WatchSource:0}: Error finding container 0b8c2ace276c4224edde4a734104fffc51b41e44360bf9a2624e86cb56d6ccc5: Status 404 returned error can't find the container with id 0b8c2ace276c4224edde4a734104fffc51b41e44360bf9a2624e86cb56d6ccc5
Jul 17 01:03:01.657533 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:01.657498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" event={"ID":"3c857f5c-c90e-4232-a24a-1c377c0062c7","Type":"ContainerStarted","Data":"0b8c2ace276c4224edde4a734104fffc51b41e44360bf9a2624e86cb56d6ccc5"}
Jul 17 01:03:02.664998 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:02.664957    2573 generic.go:356] "Generic (PLEG): container finished" podID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerID="fe4dd9bdf81f0b091e278b76e32ccecb041dd7e17b5ab8c8a309a1c3671f7d43" exitCode=6
Jul 17 01:03:02.665474 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:02.665045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" event={"ID":"3c857f5c-c90e-4232-a24a-1c377c0062c7","Type":"ContainerDied","Data":"fe4dd9bdf81f0b091e278b76e32ccecb041dd7e17b5ab8c8a309a1c3671f7d43"}
Jul 17 01:03:02.665474 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:02.665333    2573 scope.go:117] "RemoveContainer" containerID="fe4dd9bdf81f0b091e278b76e32ccecb041dd7e17b5ab8c8a309a1c3671f7d43"
Jul 17 01:03:03.302740 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.302695    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"]
Jul 17 01:03:03.303159 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.303143    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:03.303216 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.303162    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14003462-eae6-442e-ab4c-c65fb5898b38" containerName="revoke-keys"
Jul 17 01:03:03.306435 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.306411    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"
Jul 17 01:03:03.312658 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.312626    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"]
Jul 17 01:03:03.360985 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.360943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8eeda96-06c3-4744-aee8-cc61ae8dbc1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx\" (UID: \"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"
Jul 17 01:03:03.461854 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.461810    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8eeda96-06c3-4744-aee8-cc61ae8dbc1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx\" (UID: \"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"
Jul 17 01:03:03.462461 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.462437    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8eeda96-06c3-4744-aee8-cc61ae8dbc1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx\" (UID: \"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"
Jul 17 01:03:03.620269 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.620231    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"
Jul 17 01:03:03.675090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.675045    2573 generic.go:356] "Generic (PLEG): container finished" podID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerID="2a66740a8e69afbe7c31c2f969576bfe479212d861fc1217ada8f917d9663914" exitCode=6
Jul 17 01:03:03.675637 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.675193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" event={"ID":"3c857f5c-c90e-4232-a24a-1c377c0062c7","Type":"ContainerDied","Data":"2a66740a8e69afbe7c31c2f969576bfe479212d861fc1217ada8f917d9663914"}
Jul 17 01:03:03.675637 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.675282    2573 scope.go:117] "RemoveContainer" containerID="fe4dd9bdf81f0b091e278b76e32ccecb041dd7e17b5ab8c8a309a1c3671f7d43"
Jul 17 01:03:03.676269 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.676242    2573 scope.go:117] "RemoveContainer" containerID="2a66740a8e69afbe7c31c2f969576bfe479212d861fc1217ada8f917d9663914"
Jul 17 01:03:03.677157 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:03.677115    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp_odh-ai-gateway-infra(3c857f5c-c90e-4232-a24a-1c377c0062c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7"
Jul 17 01:03:03.793086 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:03.793050    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"]
Jul 17 01:03:03.794729 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:03:03.794689    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eeda96_06c3_4744_aee8_cc61ae8dbc1b.slice/crio-82c63f3310c6e54a3b90b462b77886f6b429c2e67629f1a16e0e62b3617b562e WatchSource:0}: Error finding container 82c63f3310c6e54a3b90b462b77886f6b429c2e67629f1a16e0e62b3617b562e: Status 404 returned error can't find the container with id 82c63f3310c6e54a3b90b462b77886f6b429c2e67629f1a16e0e62b3617b562e
Jul 17 01:03:04.683662 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:04.683629    2573 scope.go:117] "RemoveContainer" containerID="2a66740a8e69afbe7c31c2f969576bfe479212d861fc1217ada8f917d9663914"
Jul 17 01:03:04.684166 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:04.683917    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp_odh-ai-gateway-infra(3c857f5c-c90e-4232-a24a-1c377c0062c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7"
Jul 17 01:03:04.685253 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:04.685229    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerID="f9c2e29312c05fcafa0e516aacfb1514cfdb865bc08da01b4f4d393b59c59b26" exitCode=6
Jul 17 01:03:04.685367 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:04.685281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" event={"ID":"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b","Type":"ContainerDied","Data":"f9c2e29312c05fcafa0e516aacfb1514cfdb865bc08da01b4f4d393b59c59b26"}
Jul 17 01:03:04.685367 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:04.685321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" event={"ID":"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b","Type":"ContainerStarted","Data":"82c63f3310c6e54a3b90b462b77886f6b429c2e67629f1a16e0e62b3617b562e"}
Jul 17 01:03:04.685513 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:04.685502    2573 scope.go:117] "RemoveContainer" containerID="f9c2e29312c05fcafa0e516aacfb1514cfdb865bc08da01b4f4d393b59c59b26"
Jul 17 01:03:05.693938 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:05.693822    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerID="3beac12f7ecd6c5f26ae8b71c5bc0f7815c605545c4b0de1cdd5ac6fe58bbfd0" exitCode=6
Jul 17 01:03:05.693938 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:05.693905    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" event={"ID":"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b","Type":"ContainerDied","Data":"3beac12f7ecd6c5f26ae8b71c5bc0f7815c605545c4b0de1cdd5ac6fe58bbfd0"}
Jul 17 01:03:05.694466 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:05.693973    2573 scope.go:117] "RemoveContainer" containerID="f9c2e29312c05fcafa0e516aacfb1514cfdb865bc08da01b4f4d393b59c59b26"
Jul 17 01:03:05.694466 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:05.694191    2573 scope.go:117] "RemoveContainer" containerID="3beac12f7ecd6c5f26ae8b71c5bc0f7815c605545c4b0de1cdd5ac6fe58bbfd0"
Jul 17 01:03:05.694466 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:05.694445    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx_odh-ai-gateway-infra(f8eeda96-06c3-4744-aee8-cc61ae8dbc1b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b"
Jul 17 01:03:06.701517 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:06.701489    2573 scope.go:117] "RemoveContainer" containerID="3beac12f7ecd6c5f26ae8b71c5bc0f7815c605545c4b0de1cdd5ac6fe58bbfd0"
Jul 17 01:03:06.702095 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:06.701694    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx_odh-ai-gateway-infra(f8eeda96-06c3-4744-aee8-cc61ae8dbc1b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b"
Jul 17 01:03:09.033543 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.033445    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"]
Jul 17 01:03:09.038750 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.038711    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"
Jul 17 01:03:09.043293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.043252    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"]
Jul 17 01:03:09.110081 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.110031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e706a4d4-dec8-49b1-8aed-6334fced0b3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-zcs75\" (UID: \"e706a4d4-dec8-49b1-8aed-6334fced0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"
Jul 17 01:03:09.210877 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.210826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e706a4d4-dec8-49b1-8aed-6334fced0b3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-zcs75\" (UID: \"e706a4d4-dec8-49b1-8aed-6334fced0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"
Jul 17 01:03:09.211581 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.211550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e706a4d4-dec8-49b1-8aed-6334fced0b3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-zcs75\" (UID: \"e706a4d4-dec8-49b1-8aed-6334fced0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"
Jul 17 01:03:09.354448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.354336    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"
Jul 17 01:03:09.508805 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.508760    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"]
Jul 17 01:03:09.510516 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:03:09.510477    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode706a4d4_dec8_49b1_8aed_6334fced0b3e.slice/crio-8ca796a0413ee240d058118201adcb1af368b4b420d38dfb665146ebd9606fa4 WatchSource:0}: Error finding container 8ca796a0413ee240d058118201adcb1af368b4b420d38dfb665146ebd9606fa4: Status 404 returned error can't find the container with id 8ca796a0413ee240d058118201adcb1af368b4b420d38dfb665146ebd9606fa4
Jul 17 01:03:09.724108 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:09.724006    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" event={"ID":"e706a4d4-dec8-49b1-8aed-6334fced0b3e","Type":"ContainerStarted","Data":"8ca796a0413ee240d058118201adcb1af368b4b420d38dfb665146ebd9606fa4"}
Jul 17 01:03:10.732274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:10.732176    2573 generic.go:356] "Generic (PLEG): container finished" podID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerID="1e29edfbbbd8d52708c86203f1fbdeeec71a35ae64e09e95205da5c53fab186d" exitCode=6
Jul 17 01:03:10.732734 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:10.732267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" event={"ID":"e706a4d4-dec8-49b1-8aed-6334fced0b3e","Type":"ContainerDied","Data":"1e29edfbbbd8d52708c86203f1fbdeeec71a35ae64e09e95205da5c53fab186d"}
Jul 17 01:03:10.732734 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:10.732449    2573 scope.go:117] "RemoveContainer" containerID="1e29edfbbbd8d52708c86203f1fbdeeec71a35ae64e09e95205da5c53fab186d"
Jul 17 01:03:11.742764 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:11.742646    2573 generic.go:356] "Generic (PLEG): container finished" podID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerID="090cd634e15efacecb6f256af85c7933ee1b00b740e6ef04865c11493e2c6301" exitCode=6
Jul 17 01:03:11.742764 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:11.742698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" event={"ID":"e706a4d4-dec8-49b1-8aed-6334fced0b3e","Type":"ContainerDied","Data":"090cd634e15efacecb6f256af85c7933ee1b00b740e6ef04865c11493e2c6301"}
Jul 17 01:03:11.743366 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:11.742779    2573 scope.go:117] "RemoveContainer" containerID="1e29edfbbbd8d52708c86203f1fbdeeec71a35ae64e09e95205da5c53fab186d"
Jul 17 01:03:11.743366 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:11.743049    2573 scope.go:117] "RemoveContainer" containerID="090cd634e15efacecb6f256af85c7933ee1b00b740e6ef04865c11493e2c6301"
Jul 17 01:03:11.743477 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:11.743372    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-zcs75_odh-ai-gateway-infra(e706a4d4-dec8-49b1-8aed-6334fced0b3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e"
Jul 17 01:03:12.759906 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:12.759872    2573 scope.go:117] "RemoveContainer" containerID="090cd634e15efacecb6f256af85c7933ee1b00b740e6ef04865c11493e2c6301"
Jul 17 01:03:12.760345 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:12.760140    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-zcs75_odh-ai-gateway-infra(e706a4d4-dec8-49b1-8aed-6334fced0b3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e"
Jul 17 01:03:16.347646 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.347592    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"]
Jul 17 01:03:16.352558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.352531    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"
Jul 17 01:03:16.357633 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.357590    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"]
Jul 17 01:03:16.376498 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.376454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ae236f-ff6b-4623-a46e-94fc604d74bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-2845q\" (UID: \"15ae236f-ff6b-4623-a46e-94fc604d74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"
Jul 17 01:03:16.477818 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.477668    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ae236f-ff6b-4623-a46e-94fc604d74bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-2845q\" (UID: \"15ae236f-ff6b-4623-a46e-94fc604d74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"
Jul 17 01:03:16.478391 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.478368    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ae236f-ff6b-4623-a46e-94fc604d74bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-2845q\" (UID: \"15ae236f-ff6b-4623-a46e-94fc604d74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"
Jul 17 01:03:16.666331 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.666297    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"
Jul 17 01:03:16.829291 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:16.829253    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"]
Jul 17 01:03:16.831030 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:03:16.831001    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ae236f_ff6b_4623_a46e_94fc604d74bf.slice/crio-9140d4be638e15facf7fd3fb82a8595d45518fc011cce607dcacf0517d1dc9ce WatchSource:0}: Error finding container 9140d4be638e15facf7fd3fb82a8595d45518fc011cce607dcacf0517d1dc9ce: Status 404 returned error can't find the container with id 9140d4be638e15facf7fd3fb82a8595d45518fc011cce607dcacf0517d1dc9ce
Jul 17 01:03:17.793496 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:17.793459    2573 generic.go:356] "Generic (PLEG): container finished" podID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerID="717acd45e8b3e6c73742f0b9787a55fafcf1863efb3ffb5909b82b49ed4e32ca" exitCode=6
Jul 17 01:03:17.794058 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:17.793544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" event={"ID":"15ae236f-ff6b-4623-a46e-94fc604d74bf","Type":"ContainerDied","Data":"717acd45e8b3e6c73742f0b9787a55fafcf1863efb3ffb5909b82b49ed4e32ca"}
Jul 17 01:03:17.794058 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:17.793600    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" event={"ID":"15ae236f-ff6b-4623-a46e-94fc604d74bf","Type":"ContainerStarted","Data":"9140d4be638e15facf7fd3fb82a8595d45518fc011cce607dcacf0517d1dc9ce"}
Jul 17 01:03:17.794058 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:17.793846    2573 scope.go:117] "RemoveContainer" containerID="717acd45e8b3e6c73742f0b9787a55fafcf1863efb3ffb5909b82b49ed4e32ca"
Jul 17 01:03:17.962416 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:17.962385    2573 scope.go:117] "RemoveContainer" containerID="3beac12f7ecd6c5f26ae8b71c5bc0f7815c605545c4b0de1cdd5ac6fe58bbfd0"
Jul 17 01:03:18.801382 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.801271    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerID="00d7327c3484c7829e4c449860ce752ef9ae300ffa9f5850ace31aeed9a81b62" exitCode=6
Jul 17 01:03:18.801382 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.801342    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" event={"ID":"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b","Type":"ContainerDied","Data":"00d7327c3484c7829e4c449860ce752ef9ae300ffa9f5850ace31aeed9a81b62"}
Jul 17 01:03:18.801959 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.801455    2573 scope.go:117] "RemoveContainer" containerID="3beac12f7ecd6c5f26ae8b71c5bc0f7815c605545c4b0de1cdd5ac6fe58bbfd0"
Jul 17 01:03:18.801959 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.801909    2573 scope.go:117] "RemoveContainer" containerID="00d7327c3484c7829e4c449860ce752ef9ae300ffa9f5850ace31aeed9a81b62"
Jul 17 01:03:18.802201 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:18.802177    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx_odh-ai-gateway-infra(f8eeda96-06c3-4744-aee8-cc61ae8dbc1b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b"
Jul 17 01:03:18.803711 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.803686    2573 generic.go:356] "Generic (PLEG): container finished" podID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerID="582d809df2d21d3fb2b9a13ca19fab14c1a7e5c0a0c8edf0cbcbb95a58da451c" exitCode=6
Jul 17 01:03:18.803839 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.803726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" event={"ID":"15ae236f-ff6b-4623-a46e-94fc604d74bf","Type":"ContainerDied","Data":"582d809df2d21d3fb2b9a13ca19fab14c1a7e5c0a0c8edf0cbcbb95a58da451c"}
Jul 17 01:03:18.804031 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.804012    2573 scope.go:117] "RemoveContainer" containerID="582d809df2d21d3fb2b9a13ca19fab14c1a7e5c0a0c8edf0cbcbb95a58da451c"
Jul 17 01:03:18.804279 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:18.804257    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-2845q_odh-ai-gateway-infra(15ae236f-ff6b-4623-a46e-94fc604d74bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf"
Jul 17 01:03:18.814450 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.814413    2573 scope.go:117] "RemoveContainer" containerID="717acd45e8b3e6c73742f0b9787a55fafcf1863efb3ffb5909b82b49ed4e32ca"
Jul 17 01:03:18.962575 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:18.962541    2573 scope.go:117] "RemoveContainer" containerID="2a66740a8e69afbe7c31c2f969576bfe479212d861fc1217ada8f917d9663914"
Jul 17 01:03:19.812536 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:19.812504    2573 scope.go:117] "RemoveContainer" containerID="582d809df2d21d3fb2b9a13ca19fab14c1a7e5c0a0c8edf0cbcbb95a58da451c"
Jul 17 01:03:19.813034 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:19.812784    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-2845q_odh-ai-gateway-infra(15ae236f-ff6b-4623-a46e-94fc604d74bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf"
Jul 17 01:03:19.816385 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:19.816347    2573 generic.go:356] "Generic (PLEG): container finished" podID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerID="1b42a09915b2e64e696a1155f0f911af452bd4c88e4fbe809ccab542391832e9" exitCode=6
Jul 17 01:03:19.816586 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:19.816396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" event={"ID":"3c857f5c-c90e-4232-a24a-1c377c0062c7","Type":"ContainerDied","Data":"1b42a09915b2e64e696a1155f0f911af452bd4c88e4fbe809ccab542391832e9"}
Jul 17 01:03:19.816586 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:19.816460    2573 scope.go:117] "RemoveContainer" containerID="2a66740a8e69afbe7c31c2f969576bfe479212d861fc1217ada8f917d9663914"
Jul 17 01:03:19.816870 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:19.816850    2573 scope.go:117] "RemoveContainer" containerID="1b42a09915b2e64e696a1155f0f911af452bd4c88e4fbe809ccab542391832e9"
Jul 17 01:03:19.817196 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:19.817160    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp_odh-ai-gateway-infra(3c857f5c-c90e-4232-a24a-1c377c0062c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7"
Jul 17 01:03:19.824827 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:19.824776    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"]
Jul 17 01:03:19.981001 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:19.980964    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"
Jul 17 01:03:20.012972 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.012935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8eeda96-06c3-4744-aee8-cc61ae8dbc1b-maas-api-service-ca\") pod \"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b\" (UID: \"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b\") "
Jul 17 01:03:20.013476 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.013435    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eeda96-06c3-4744-aee8-cc61ae8dbc1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" (UID: "f8eeda96-06c3-4744-aee8-cc61ae8dbc1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:03:20.114011 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.113969    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8eeda96-06c3-4744-aee8-cc61ae8dbc1b-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:20.826568 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.826526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx" event={"ID":"f8eeda96-06c3-4744-aee8-cc61ae8dbc1b","Type":"ContainerDied","Data":"82c63f3310c6e54a3b90b462b77886f6b429c2e67629f1a16e0e62b3617b562e"}
Jul 17 01:03:20.826568 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.826547    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"
Jul 17 01:03:20.827234 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.826601    2573 scope.go:117] "RemoveContainer" containerID="00d7327c3484c7829e4c449860ce752ef9ae300ffa9f5850ace31aeed9a81b62"
Jul 17 01:03:20.850643 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.850603    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"]
Jul 17 01:03:20.853927 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.853891    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"]
Jul 17 01:03:20.856941 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:20.856904    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-nzdgx"]
Jul 17 01:03:21.010954 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.010920    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"
Jul 17 01:03:21.124409 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.124307    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c857f5c-c90e-4232-a24a-1c377c0062c7-maas-api-service-ca\") pod \"3c857f5c-c90e-4232-a24a-1c377c0062c7\" (UID: \"3c857f5c-c90e-4232-a24a-1c377c0062c7\") "
Jul 17 01:03:21.124720 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.124698    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c857f5c-c90e-4232-a24a-1c377c0062c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c857f5c-c90e-4232-a24a-1c377c0062c7" (UID: "3c857f5c-c90e-4232-a24a-1c377c0062c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:03:21.225283 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.225235    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c857f5c-c90e-4232-a24a-1c377c0062c7-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:21.837829 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.837775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp" event={"ID":"3c857f5c-c90e-4232-a24a-1c377c0062c7","Type":"ContainerDied","Data":"0b8c2ace276c4224edde4a734104fffc51b41e44360bf9a2624e86cb56d6ccc5"}
Jul 17 01:03:21.838353 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.837855    2573 scope.go:117] "RemoveContainer" containerID="1b42a09915b2e64e696a1155f0f911af452bd4c88e4fbe809ccab542391832e9"
Jul 17 01:03:21.838353 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.837782    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"
Jul 17 01:03:21.860961 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.860846    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"]
Jul 17 01:03:21.863532 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.863501    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-jrkgp"]
Jul 17 01:03:21.968869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.968830    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" path="/var/lib/kubelet/pods/3c857f5c-c90e-4232-a24a-1c377c0062c7/volumes"
Jul 17 01:03:21.969293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:21.969276    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" path="/var/lib/kubelet/pods/f8eeda96-06c3-4744-aee8-cc61ae8dbc1b/volumes"
Jul 17 01:03:23.962271 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:23.962229    2573 scope.go:117] "RemoveContainer" containerID="090cd634e15efacecb6f256af85c7933ee1b00b740e6ef04865c11493e2c6301"
Jul 17 01:03:24.861293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:24.861248    2573 generic.go:356] "Generic (PLEG): container finished" podID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerID="8a75317a2a654e49c23c2dbffcc23d916e081d702ddd352fb028035dd2abeb1b" exitCode=6
Jul 17 01:03:24.861503 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:24.861322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" event={"ID":"e706a4d4-dec8-49b1-8aed-6334fced0b3e","Type":"ContainerDied","Data":"8a75317a2a654e49c23c2dbffcc23d916e081d702ddd352fb028035dd2abeb1b"}
Jul 17 01:03:24.861503 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:24.861424    2573 scope.go:117] "RemoveContainer" containerID="090cd634e15efacecb6f256af85c7933ee1b00b740e6ef04865c11493e2c6301"
Jul 17 01:03:24.861874 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:24.861834    2573 scope.go:117] "RemoveContainer" containerID="8a75317a2a654e49c23c2dbffcc23d916e081d702ddd352fb028035dd2abeb1b"
Jul 17 01:03:24.862158 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:24.862138    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-zcs75_odh-ai-gateway-infra(e706a4d4-dec8-49b1-8aed-6334fced0b3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e"
Jul 17 01:03:25.885113 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:25.885060    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"]
Jul 17 01:03:26.037957 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.037916    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"
Jul 17 01:03:26.068505 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.068467    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e706a4d4-dec8-49b1-8aed-6334fced0b3e-maas-api-service-ca\") pod \"e706a4d4-dec8-49b1-8aed-6334fced0b3e\" (UID: \"e706a4d4-dec8-49b1-8aed-6334fced0b3e\") "
Jul 17 01:03:26.068928 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.068891    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706a4d4-dec8-49b1-8aed-6334fced0b3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e706a4d4-dec8-49b1-8aed-6334fced0b3e" (UID: "e706a4d4-dec8-49b1-8aed-6334fced0b3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:03:26.170323 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.170197    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e706a4d4-dec8-49b1-8aed-6334fced0b3e-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:26.877247 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.877214    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"
Jul 17 01:03:26.877247 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.877242    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75" event={"ID":"e706a4d4-dec8-49b1-8aed-6334fced0b3e","Type":"ContainerDied","Data":"8ca796a0413ee240d058118201adcb1af368b4b420d38dfb665146ebd9606fa4"}
Jul 17 01:03:26.877508 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.877308    2573 scope.go:117] "RemoveContainer" containerID="8a75317a2a654e49c23c2dbffcc23d916e081d702ddd352fb028035dd2abeb1b"
Jul 17 01:03:26.900502 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.900458    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"]
Jul 17 01:03:26.906623 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:26.906571    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-zcs75"]
Jul 17 01:03:27.969735 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:27.969694    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" path="/var/lib/kubelet/pods/e706a4d4-dec8-49b1-8aed-6334fced0b3e/volumes"
Jul 17 01:03:30.963105 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:30.963060    2573 scope.go:117] "RemoveContainer" containerID="582d809df2d21d3fb2b9a13ca19fab14c1a7e5c0a0c8edf0cbcbb95a58da451c"
Jul 17 01:03:31.912328 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:31.912227    2573 generic.go:356] "Generic (PLEG): container finished" podID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerID="1713715c3ff172960c26a7eca0835fca6d705dc9c5940f6a430e94e6ac47e178" exitCode=6
Jul 17 01:03:31.912328 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:31.912308    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" event={"ID":"15ae236f-ff6b-4623-a46e-94fc604d74bf","Type":"ContainerDied","Data":"1713715c3ff172960c26a7eca0835fca6d705dc9c5940f6a430e94e6ac47e178"}
Jul 17 01:03:31.912567 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:31.912372    2573 scope.go:117] "RemoveContainer" containerID="582d809df2d21d3fb2b9a13ca19fab14c1a7e5c0a0c8edf0cbcbb95a58da451c"
Jul 17 01:03:31.912766 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:31.912750    2573 scope.go:117] "RemoveContainer" containerID="1713715c3ff172960c26a7eca0835fca6d705dc9c5940f6a430e94e6ac47e178"
Jul 17 01:03:31.913044 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:31.913016    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-2845q_odh-ai-gateway-infra(15ae236f-ff6b-4623-a46e-94fc604d74bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf"
Jul 17 01:03:32.935994 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:32.935949    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"]
Jul 17 01:03:33.086554 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.086520    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"
Jul 17 01:03:33.138065 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.138030    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ae236f-ff6b-4623-a46e-94fc604d74bf-maas-api-service-ca\") pod \"15ae236f-ff6b-4623-a46e-94fc604d74bf\" (UID: \"15ae236f-ff6b-4623-a46e-94fc604d74bf\") "
Jul 17 01:03:33.138522 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.138494    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ae236f-ff6b-4623-a46e-94fc604d74bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15ae236f-ff6b-4623-a46e-94fc604d74bf" (UID: "15ae236f-ff6b-4623-a46e-94fc604d74bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:03:33.239716 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.239607    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ae236f-ff6b-4623-a46e-94fc604d74bf-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:33.928037 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.928006    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"
Jul 17 01:03:33.928280 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.928033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q" event={"ID":"15ae236f-ff6b-4623-a46e-94fc604d74bf","Type":"ContainerDied","Data":"9140d4be638e15facf7fd3fb82a8595d45518fc011cce607dcacf0517d1dc9ce"}
Jul 17 01:03:33.928280 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.928096    2573 scope.go:117] "RemoveContainer" containerID="1713715c3ff172960c26a7eca0835fca6d705dc9c5940f6a430e94e6ac47e178"
Jul 17 01:03:33.952358 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.952315    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"]
Jul 17 01:03:33.958111 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.958070    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-2845q"]
Jul 17 01:03:33.968620 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:33.968585    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" path="/var/lib/kubelet/pods/15ae236f-ff6b-4623-a46e-94fc604d74bf/volumes"
Jul 17 01:03:34.506693 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.506646    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"]
Jul 17 01:03:34.507091 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507077    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507094    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507108    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:34.507142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507116    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:34.507142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507128    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:34.507142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507135    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507148    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507153    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507163    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507168    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507176    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507181    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507215    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507221    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507227    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507233    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507239    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507244    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507254    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:34.507294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507259    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507319    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507329    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507336    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507343    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507348    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507355    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507360    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507365    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507370    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706a4d4-dec8-49b1-8aed-6334fced0b3e" containerName="revoke-keys"
Jul 17 01:03:34.507688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.507376    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:34.511957 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.511929    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"
Jul 17 01:03:34.514001 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.513963    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:03:34.517315 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.517280    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"]
Jul 17 01:03:34.651156 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.651114    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bcf850-8ef9-4069-8b1e-8f391513b3b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27f370-m56kr\" (UID: \"e5bcf850-8ef9-4069-8b1e-8f391513b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"
Jul 17 01:03:34.751926 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.751882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bcf850-8ef9-4069-8b1e-8f391513b3b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27f370-m56kr\" (UID: \"e5bcf850-8ef9-4069-8b1e-8f391513b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"
Jul 17 01:03:34.752594 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.752571    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bcf850-8ef9-4069-8b1e-8f391513b3b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27f370-m56kr\" (UID: \"e5bcf850-8ef9-4069-8b1e-8f391513b3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"
Jul 17 01:03:34.826423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:34.826328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"
Jul 17 01:03:35.014709 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:35.014670    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"]
Jul 17 01:03:35.017833 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:03:35.017781    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bcf850_8ef9_4069_8b1e_8f391513b3b4.slice/crio-a0b9136aea96bab79add5ef0c1895f1089022217911dbc7bd6dfe9ddd87d9f82 WatchSource:0}: Error finding container a0b9136aea96bab79add5ef0c1895f1089022217911dbc7bd6dfe9ddd87d9f82: Status 404 returned error can't find the container with id a0b9136aea96bab79add5ef0c1895f1089022217911dbc7bd6dfe9ddd87d9f82
Jul 17 01:03:35.945814 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:35.945758    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr" event={"ID":"e5bcf850-8ef9-4069-8b1e-8f391513b3b4","Type":"ContainerStarted","Data":"60dc22e3307cb3cc248c02c7709f144ec7f0492cd1128ac455c0cbcfe52a6239"}
Jul 17 01:03:35.946039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:35.945842    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr" event={"ID":"e5bcf850-8ef9-4069-8b1e-8f391513b3b4","Type":"ContainerStarted","Data":"a0b9136aea96bab79add5ef0c1895f1089022217911dbc7bd6dfe9ddd87d9f82"}
Jul 17 01:03:35.961194 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:35.961120    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr" podStartSLOduration=1.961095528 podStartE2EDuration="1.961095528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:03:35.958121391 +0000 UTC m=+2810.534273965" watchObservedRunningTime="2026-07-17 01:03:35.961095528 +0000 UTC m=+2810.537248102"
Jul 17 01:03:36.955199 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:36.955149    2573 generic.go:356] "Generic (PLEG): container finished" podID="e5bcf850-8ef9-4069-8b1e-8f391513b3b4" containerID="60dc22e3307cb3cc248c02c7709f144ec7f0492cd1128ac455c0cbcfe52a6239" exitCode=0
Jul 17 01:03:36.955742 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:36.955228    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr" event={"ID":"e5bcf850-8ef9-4069-8b1e-8f391513b3b4","Type":"ContainerDied","Data":"60dc22e3307cb3cc248c02c7709f144ec7f0492cd1128ac455c0cbcfe52a6239"}
Jul 17 01:03:38.112611 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:38.112578    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"
Jul 17 01:03:38.187644 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:38.187601    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bcf850-8ef9-4069-8b1e-8f391513b3b4-maas-api-service-ca\") pod \"e5bcf850-8ef9-4069-8b1e-8f391513b3b4\" (UID: \"e5bcf850-8ef9-4069-8b1e-8f391513b3b4\") "
Jul 17 01:03:38.188024 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:38.188001    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bcf850-8ef9-4069-8b1e-8f391513b3b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5bcf850-8ef9-4069-8b1e-8f391513b3b4" (UID: "e5bcf850-8ef9-4069-8b1e-8f391513b3b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:03:38.289292 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:38.289250    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bcf850-8ef9-4069-8b1e-8f391513b3b4-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:38.969002 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:38.968968    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr"
Jul 17 01:03:38.969002 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:38.969000    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27f370-m56kr" event={"ID":"e5bcf850-8ef9-4069-8b1e-8f391513b3b4","Type":"ContainerDied","Data":"a0b9136aea96bab79add5ef0c1895f1089022217911dbc7bd6dfe9ddd87d9f82"}
Jul 17 01:03:38.969259 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:38.969058    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0b9136aea96bab79add5ef0c1895f1089022217911dbc7bd6dfe9ddd87d9f82"
Jul 17 01:03:44.942504 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:44.942449    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"]
Jul 17 01:03:44.943110 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:44.942767    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8" podUID="566dec08-45d8-4b11-9f95-45e2341ed0a4" containerName="maas-api" containerID="cri-o://ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428" gracePeriod=30
Jul 17 01:03:45.226893 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.224994    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:03:45.355803 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.355740    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqmw5\" (UniqueName: \"kubernetes.io/projected/566dec08-45d8-4b11-9f95-45e2341ed0a4-kube-api-access-dqmw5\") pod \"566dec08-45d8-4b11-9f95-45e2341ed0a4\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") "
Jul 17 01:03:45.356039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.355912    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls\") pod \"566dec08-45d8-4b11-9f95-45e2341ed0a4\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") "
Jul 17 01:03:45.356039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.355956    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566dec08-45d8-4b11-9f95-45e2341ed0a4-openshift-service-ca\") pod \"566dec08-45d8-4b11-9f95-45e2341ed0a4\" (UID: \"566dec08-45d8-4b11-9f95-45e2341ed0a4\") "
Jul 17 01:03:45.356381 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.356354    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566dec08-45d8-4b11-9f95-45e2341ed0a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "566dec08-45d8-4b11-9f95-45e2341ed0a4" (UID: "566dec08-45d8-4b11-9f95-45e2341ed0a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:03:45.358238 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.358205    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "566dec08-45d8-4b11-9f95-45e2341ed0a4" (UID: "566dec08-45d8-4b11-9f95-45e2341ed0a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 01:03:45.358238 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.358211    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566dec08-45d8-4b11-9f95-45e2341ed0a4-kube-api-access-dqmw5" (OuterVolumeSpecName: "kube-api-access-dqmw5") pod "566dec08-45d8-4b11-9f95-45e2341ed0a4" (UID: "566dec08-45d8-4b11-9f95-45e2341ed0a4"). InnerVolumeSpecName "kube-api-access-dqmw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:03:45.457017 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.456897    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566dec08-45d8-4b11-9f95-45e2341ed0a4-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:45.457017 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.456940    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566dec08-45d8-4b11-9f95-45e2341ed0a4-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:45.457017 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:45.456952    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqmw5\" (UniqueName: \"kubernetes.io/projected/566dec08-45d8-4b11-9f95-45e2341ed0a4-kube-api-access-dqmw5\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:03:46.014497 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.014455    2573 generic.go:356] "Generic (PLEG): container finished" podID="566dec08-45d8-4b11-9f95-45e2341ed0a4" containerID="ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428" exitCode=0
Jul 17 01:03:46.014497 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.014501    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"
Jul 17 01:03:46.015136 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.014527    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8" event={"ID":"566dec08-45d8-4b11-9f95-45e2341ed0a4","Type":"ContainerDied","Data":"ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428"}
Jul 17 01:03:46.015136 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.014583    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8" event={"ID":"566dec08-45d8-4b11-9f95-45e2341ed0a4","Type":"ContainerDied","Data":"3f781c7e1b700516e232b84a0bc2d69203cd1d025f93f6b20de6ccdb65334aed"}
Jul 17 01:03:46.015136 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.014615    2573 scope.go:117] "RemoveContainer" containerID="ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428"
Jul 17 01:03:46.024897 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.024874    2573 scope.go:117] "RemoveContainer" containerID="ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428"
Jul 17 01:03:46.025296 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:46.025271    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428\": container with ID starting with ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428 not found: ID does not exist" containerID="ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428"
Jul 17 01:03:46.025399 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.025306    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428"} err="failed to get container status \"ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428\": rpc error: code = NotFound desc = could not find container \"ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428\": container with ID starting with ef873c815de94b8d5ed3cbbb1621f171aff66f89d5d5893f55781ed487651428 not found: ID does not exist"
Jul 17 01:03:46.030955 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.030870    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"]
Jul 17 01:03:46.033804 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:46.033749    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27f370-76546579b8-x92q8"]
Jul 17 01:03:47.969115 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:47.969071    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566dec08-45d8-4b11-9f95-45e2341ed0a4" path="/var/lib/kubelet/pods/566dec08-45d8-4b11-9f95-45e2341ed0a4/volumes"
Jul 17 01:03:53.346556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.346495    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"]
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.346940    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5bcf850-8ef9-4069-8b1e-8f391513b3b4" containerName="revoke-keys"
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.346957    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5bcf850-8ef9-4069-8b1e-8f391513b3b4" containerName="revoke-keys"
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.346970    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566dec08-45d8-4b11-9f95-45e2341ed0a4" containerName="maas-api"
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.346976    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="566dec08-45d8-4b11-9f95-45e2341ed0a4" containerName="maas-api"
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.346996    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.347002    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.347013    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:53.347039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.347019    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c857f5c-c90e-4232-a24a-1c377c0062c7" containerName="revoke-keys"
Jul 17 01:03:53.347301 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.347077    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ae236f-ff6b-4623-a46e-94fc604d74bf" containerName="revoke-keys"
Jul 17 01:03:53.347301 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.347086    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5bcf850-8ef9-4069-8b1e-8f391513b3b4" containerName="revoke-keys"
Jul 17 01:03:53.347301 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.347093    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="566dec08-45d8-4b11-9f95-45e2341ed0a4" containerName="maas-api"
Jul 17 01:03:53.347301 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.347099    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8eeda96-06c3-4744-aee8-cc61ae8dbc1b" containerName="revoke-keys"
Jul 17 01:03:53.351528 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.351502    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"
Jul 17 01:03:53.353785 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.353753    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:03:53.356984 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.356941    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"]
Jul 17 01:03:53.538364 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.538320    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca902f-c507-473b-bb6b-17b4a42249c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9\" (UID: \"52ca902f-c507-473b-bb6b-17b4a42249c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"
Jul 17 01:03:53.639633 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.639508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca902f-c507-473b-bb6b-17b4a42249c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9\" (UID: \"52ca902f-c507-473b-bb6b-17b4a42249c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"
Jul 17 01:03:53.640272 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.640245    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca902f-c507-473b-bb6b-17b4a42249c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9\" (UID: \"52ca902f-c507-473b-bb6b-17b4a42249c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"
Jul 17 01:03:53.665487 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.665417    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"
Jul 17 01:03:53.821688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:53.821658    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"]
Jul 17 01:03:53.823864 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:03:53.823834    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ca902f_c507_473b_bb6b_17b4a42249c9.slice/crio-aff6f2fafce5dd19459039af3dda898f9355e3bb0a627e4118e43ac337eedd0c WatchSource:0}: Error finding container aff6f2fafce5dd19459039af3dda898f9355e3bb0a627e4118e43ac337eedd0c: Status 404 returned error can't find the container with id aff6f2fafce5dd19459039af3dda898f9355e3bb0a627e4118e43ac337eedd0c
Jul 17 01:03:54.066435 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:54.066403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" event={"ID":"52ca902f-c507-473b-bb6b-17b4a42249c9","Type":"ContainerStarted","Data":"aff6f2fafce5dd19459039af3dda898f9355e3bb0a627e4118e43ac337eedd0c"}
Jul 17 01:03:55.074529 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:55.074493    2573 generic.go:356] "Generic (PLEG): container finished" podID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerID="91b2e641ac1cb4315f04143ae5ee15dd0e81bb17d1e256e5ea40145b6e0794bb" exitCode=6
Jul 17 01:03:55.075108 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:55.074580    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" event={"ID":"52ca902f-c507-473b-bb6b-17b4a42249c9","Type":"ContainerDied","Data":"91b2e641ac1cb4315f04143ae5ee15dd0e81bb17d1e256e5ea40145b6e0794bb"}
Jul 17 01:03:55.075108 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:55.074784    2573 scope.go:117] "RemoveContainer" containerID="91b2e641ac1cb4315f04143ae5ee15dd0e81bb17d1e256e5ea40145b6e0794bb"
Jul 17 01:03:56.083753 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:56.083709    2573 generic.go:356] "Generic (PLEG): container finished" podID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerID="273ce39c9d6a35bc157264f5d5e6f1958505c65a31696c0b6679d389a6f5cf70" exitCode=6
Jul 17 01:03:56.084288 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:56.083820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" event={"ID":"52ca902f-c507-473b-bb6b-17b4a42249c9","Type":"ContainerDied","Data":"273ce39c9d6a35bc157264f5d5e6f1958505c65a31696c0b6679d389a6f5cf70"}
Jul 17 01:03:56.084288 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:56.083908    2573 scope.go:117] "RemoveContainer" containerID="91b2e641ac1cb4315f04143ae5ee15dd0e81bb17d1e256e5ea40145b6e0794bb"
Jul 17 01:03:56.084288 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:56.084130    2573 scope.go:117] "RemoveContainer" containerID="273ce39c9d6a35bc157264f5d5e6f1958505c65a31696c0b6679d389a6f5cf70"
Jul 17 01:03:56.084469 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:56.084407    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9_odh-ai-gateway-infra(52ca902f-c507-473b-bb6b-17b4a42249c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9"
Jul 17 01:03:57.091423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:57.091389    2573 scope.go:117] "RemoveContainer" containerID="273ce39c9d6a35bc157264f5d5e6f1958505c65a31696c0b6679d389a6f5cf70"
Jul 17 01:03:57.091991 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:03:57.091646    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9_odh-ai-gateway-infra(52ca902f-c507-473b-bb6b-17b4a42249c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9"
Jul 17 01:03:59.071722 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.071672    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"]
Jul 17 01:03:59.076969 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.076931    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"
Jul 17 01:03:59.081073 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.081030    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"]
Jul 17 01:03:59.089318 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.089282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bf630a-14a1-412c-8e41-0298e727f002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq\" (UID: \"b9bf630a-14a1-412c-8e41-0298e727f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"
Jul 17 01:03:59.190096 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.190048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bf630a-14a1-412c-8e41-0298e727f002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq\" (UID: \"b9bf630a-14a1-412c-8e41-0298e727f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"
Jul 17 01:03:59.190846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.190778    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bf630a-14a1-412c-8e41-0298e727f002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq\" (UID: \"b9bf630a-14a1-412c-8e41-0298e727f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"
Jul 17 01:03:59.391682 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.391560    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"
Jul 17 01:03:59.551901 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:03:59.551668    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"]
Jul 17 01:03:59.554389 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:03:59.554343    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bf630a_14a1_412c_8e41_0298e727f002.slice/crio-bdd5facdc3bb873748b1240e45b8bb915f3651412957a545b566b0dd4bc5d8e3 WatchSource:0}: Error finding container bdd5facdc3bb873748b1240e45b8bb915f3651412957a545b566b0dd4bc5d8e3: Status 404 returned error can't find the container with id bdd5facdc3bb873748b1240e45b8bb915f3651412957a545b566b0dd4bc5d8e3
Jul 17 01:04:00.117187 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.117135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" event={"ID":"b9bf630a-14a1-412c-8e41-0298e727f002","Type":"ContainerStarted","Data":"8d7d450cc434f0d17e9f7d44eb0bdbc36bb8c449b655c0ec0034b83ff1952211"}
Jul 17 01:04:00.117737 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.117216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" event={"ID":"b9bf630a-14a1-412c-8e41-0298e727f002","Type":"ContainerStarted","Data":"bdd5facdc3bb873748b1240e45b8bb915f3651412957a545b566b0dd4bc5d8e3"}
Jul 17 01:04:00.132941 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.132863    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" podStartSLOduration=1.1328395119999999 podStartE2EDuration="1.132839512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:04:00.129951006 +0000 UTC m=+2834.706103581" watchObservedRunningTime="2026-07-17 01:04:00.132839512 +0000 UTC m=+2834.708992085"
Jul 17 01:04:00.667506 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.667443    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"]
Jul 17 01:04:00.671239 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.671202    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"
Jul 17 01:04:00.677474 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.677423    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"]
Jul 17 01:04:00.705883 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.705728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2856d5b3-ccc6-4247-b9ef-75078e6c3c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2\" (UID: \"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"
Jul 17 01:04:00.806716 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.806666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2856d5b3-ccc6-4247-b9ef-75078e6c3c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2\" (UID: \"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"
Jul 17 01:04:00.807460 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.807430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2856d5b3-ccc6-4247-b9ef-75078e6c3c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2\" (UID: \"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"
Jul 17 01:04:00.913241 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.913189    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"]
Jul 17 01:04:00.917469 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.917438    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"
Jul 17 01:04:00.923770 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.923674    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"]
Jul 17 01:04:00.986219 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:00.986100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"
Jul 17 01:04:01.008998 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.008945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8836e04-b365-439f-9057-4493d2ebf5df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99\" (UID: \"f8836e04-b365-439f-9057-4493d2ebf5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"
Jul 17 01:04:01.110588 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.110546    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8836e04-b365-439f-9057-4493d2ebf5df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99\" (UID: \"f8836e04-b365-439f-9057-4493d2ebf5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"
Jul 17 01:04:01.111318 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.111293    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8836e04-b365-439f-9057-4493d2ebf5df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99\" (UID: \"f8836e04-b365-439f-9057-4493d2ebf5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"
Jul 17 01:04:01.126225 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.126178    2573 generic.go:356] "Generic (PLEG): container finished" podID="b9bf630a-14a1-412c-8e41-0298e727f002" containerID="8d7d450cc434f0d17e9f7d44eb0bdbc36bb8c449b655c0ec0034b83ff1952211" exitCode=6
Jul 17 01:04:01.126745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.126260    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" event={"ID":"b9bf630a-14a1-412c-8e41-0298e727f002","Type":"ContainerDied","Data":"8d7d450cc434f0d17e9f7d44eb0bdbc36bb8c449b655c0ec0034b83ff1952211"}
Jul 17 01:04:01.126745 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.126572    2573 scope.go:117] "RemoveContainer" containerID="8d7d450cc434f0d17e9f7d44eb0bdbc36bb8c449b655c0ec0034b83ff1952211"
Jul 17 01:04:01.140734 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.140690    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"]
Jul 17 01:04:01.143653 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:04:01.143591    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2856d5b3_ccc6_4247_b9ef_75078e6c3c1d.slice/crio-2c618b17e7fe439f2f9f118a4258d9f21e56f7b6d74047818c889a656754a0d8 WatchSource:0}: Error finding container 2c618b17e7fe439f2f9f118a4258d9f21e56f7b6d74047818c889a656754a0d8: Status 404 returned error can't find the container with id 2c618b17e7fe439f2f9f118a4258d9f21e56f7b6d74047818c889a656754a0d8
Jul 17 01:04:01.231470 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.231427    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"
Jul 17 01:04:01.396965 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:01.396932    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"]
Jul 17 01:04:01.399063 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:04:01.399027    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8836e04_b365_439f_9057_4493d2ebf5df.slice/crio-3bfaa5407e9d13bc73b50090a193c5e130b20a6b62df4cf148c32259965f490c WatchSource:0}: Error finding container 3bfaa5407e9d13bc73b50090a193c5e130b20a6b62df4cf148c32259965f490c: Status 404 returned error can't find the container with id 3bfaa5407e9d13bc73b50090a193c5e130b20a6b62df4cf148c32259965f490c
Jul 17 01:04:02.134071 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.134036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" event={"ID":"f8836e04-b365-439f-9057-4493d2ebf5df","Type":"ContainerStarted","Data":"e2b124af16e41f271c8568ab42b6de54d65c6a68b376e37364a136f04d788f63"}
Jul 17 01:04:02.134580 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.134107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" event={"ID":"f8836e04-b365-439f-9057-4493d2ebf5df","Type":"ContainerStarted","Data":"3bfaa5407e9d13bc73b50090a193c5e130b20a6b62df4cf148c32259965f490c"}
Jul 17 01:04:02.136080 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.136047    2573 generic.go:356] "Generic (PLEG): container finished" podID="b9bf630a-14a1-412c-8e41-0298e727f002" containerID="ff360fb627e0f8396cdf19daa91a36b24051b81b5c44af5dedf5eec321f5fb96" exitCode=6
Jul 17 01:04:02.136224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.136118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" event={"ID":"b9bf630a-14a1-412c-8e41-0298e727f002","Type":"ContainerDied","Data":"ff360fb627e0f8396cdf19daa91a36b24051b81b5c44af5dedf5eec321f5fb96"}
Jul 17 01:04:02.136224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.136184    2573 scope.go:117] "RemoveContainer" containerID="8d7d450cc434f0d17e9f7d44eb0bdbc36bb8c449b655c0ec0034b83ff1952211"
Jul 17 01:04:02.136395 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.136382    2573 scope.go:117] "RemoveContainer" containerID="ff360fb627e0f8396cdf19daa91a36b24051b81b5c44af5dedf5eec321f5fb96"
Jul 17 01:04:02.136649 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:02.136625    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq_odh-ai-gateway-infra(b9bf630a-14a1-412c-8e41-0298e727f002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" podUID="b9bf630a-14a1-412c-8e41-0298e727f002"
Jul 17 01:04:02.138874 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.138845    2573 generic.go:356] "Generic (PLEG): container finished" podID="2856d5b3-ccc6-4247-b9ef-75078e6c3c1d" containerID="5ecd676fceafba7c8b99a828dfba2ba31f8c7fbc903c0518d7e65af480aa1812" exitCode=0
Jul 17 01:04:02.139022 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.138915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2" event={"ID":"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d","Type":"ContainerDied","Data":"5ecd676fceafba7c8b99a828dfba2ba31f8c7fbc903c0518d7e65af480aa1812"}
Jul 17 01:04:02.139022 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.138963    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2" event={"ID":"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d","Type":"ContainerStarted","Data":"2c618b17e7fe439f2f9f118a4258d9f21e56f7b6d74047818c889a656754a0d8"}
Jul 17 01:04:02.149230 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:02.149167    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" podStartSLOduration=2.149147181 podStartE2EDuration="2.149147181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:04:02.14746106 +0000 UTC m=+2836.723613634" watchObservedRunningTime="2026-07-17 01:04:02.149147181 +0000 UTC m=+2836.725299754"
Jul 17 01:04:03.147632 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.147594    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8836e04-b365-439f-9057-4493d2ebf5df" containerID="e2b124af16e41f271c8568ab42b6de54d65c6a68b376e37364a136f04d788f63" exitCode=6
Jul 17 01:04:03.148176 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.147693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" event={"ID":"f8836e04-b365-439f-9057-4493d2ebf5df","Type":"ContainerDied","Data":"e2b124af16e41f271c8568ab42b6de54d65c6a68b376e37364a136f04d788f63"}
Jul 17 01:04:03.148176 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.147988    2573 scope.go:117] "RemoveContainer" containerID="e2b124af16e41f271c8568ab42b6de54d65c6a68b376e37364a136f04d788f63"
Jul 17 01:04:03.150104 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.150076    2573 scope.go:117] "RemoveContainer" containerID="ff360fb627e0f8396cdf19daa91a36b24051b81b5c44af5dedf5eec321f5fb96"
Jul 17 01:04:03.150381 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:03.150352    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq_odh-ai-gateway-infra(b9bf630a-14a1-412c-8e41-0298e727f002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" podUID="b9bf630a-14a1-412c-8e41-0298e727f002"
Jul 17 01:04:03.306443 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.306405    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"
Jul 17 01:04:03.333147 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.333101    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2856d5b3-ccc6-4247-b9ef-75078e6c3c1d-maas-api-service-ca\") pod \"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d\" (UID: \"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d\") "
Jul 17 01:04:03.333485 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.333463    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2856d5b3-ccc6-4247-b9ef-75078e6c3c1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2856d5b3-ccc6-4247-b9ef-75078e6c3c1d" (UID: "2856d5b3-ccc6-4247-b9ef-75078e6c3c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:04:03.434721 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:03.434590    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2856d5b3-ccc6-4247-b9ef-75078e6c3c1d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:04.157942 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:04.157825    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8836e04-b365-439f-9057-4493d2ebf5df" containerID="f105918b04aa8df64ddea0705894a32cb936abbe2d71f75f8a64d04002b0499f" exitCode=6
Jul 17 01:04:04.157942 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:04.157917    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" event={"ID":"f8836e04-b365-439f-9057-4493d2ebf5df","Type":"ContainerDied","Data":"f105918b04aa8df64ddea0705894a32cb936abbe2d71f75f8a64d04002b0499f"}
Jul 17 01:04:04.158477 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:04.157996    2573 scope.go:117] "RemoveContainer" containerID="e2b124af16e41f271c8568ab42b6de54d65c6a68b376e37364a136f04d788f63"
Jul 17 01:04:04.158477 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:04.158215    2573 scope.go:117] "RemoveContainer" containerID="f105918b04aa8df64ddea0705894a32cb936abbe2d71f75f8a64d04002b0499f"
Jul 17 01:04:04.158477 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:04.158464    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99_odh-ai-gateway-infra(f8836e04-b365-439f-9057-4493d2ebf5df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" podUID="f8836e04-b365-439f-9057-4493d2ebf5df"
Jul 17 01:04:04.159631 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:04.159612    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2"
Jul 17 01:04:04.159763 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:04.159607    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2" event={"ID":"2856d5b3-ccc6-4247-b9ef-75078e6c3c1d","Type":"ContainerDied","Data":"2c618b17e7fe439f2f9f118a4258d9f21e56f7b6d74047818c889a656754a0d8"}
Jul 17 01:04:04.159763 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:04.159744    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c618b17e7fe439f2f9f118a4258d9f21e56f7b6d74047818c889a656754a0d8"
Jul 17 01:04:05.167760 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:05.167723    2573 scope.go:117] "RemoveContainer" containerID="f105918b04aa8df64ddea0705894a32cb936abbe2d71f75f8a64d04002b0499f"
Jul 17 01:04:05.168233 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:05.168003    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99_odh-ai-gateway-infra(f8836e04-b365-439f-9057-4493d2ebf5df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" podUID="f8836e04-b365-439f-9057-4493d2ebf5df"
Jul 17 01:04:06.388008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.387972    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"]
Jul 17 01:04:06.388455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.388398    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2856d5b3-ccc6-4247-b9ef-75078e6c3c1d" containerName="revoke-keys"
Jul 17 01:04:06.388455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.388410    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2856d5b3-ccc6-4247-b9ef-75078e6c3c1d" containerName="revoke-keys"
Jul 17 01:04:06.388550 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.388473    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2856d5b3-ccc6-4247-b9ef-75078e6c3c1d" containerName="revoke-keys"
Jul 17 01:04:06.393251 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.393202    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"
Jul 17 01:04:06.397600 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.397558    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"]
Jul 17 01:04:06.469352 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.469301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8812f2eb-3b36-4677-a6a4-fccb4be01aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-xtnbb\" (UID: \"8812f2eb-3b36-4677-a6a4-fccb4be01aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"
Jul 17 01:04:06.570730 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.570688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8812f2eb-3b36-4677-a6a4-fccb4be01aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-xtnbb\" (UID: \"8812f2eb-3b36-4677-a6a4-fccb4be01aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"
Jul 17 01:04:06.571437 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.571408    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8812f2eb-3b36-4677-a6a4-fccb4be01aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-xtnbb\" (UID: \"8812f2eb-3b36-4677-a6a4-fccb4be01aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"
Jul 17 01:04:06.707276 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.707156    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"
Jul 17 01:04:06.864272 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:06.864238    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"]
Jul 17 01:04:06.867406 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:04:06.867370    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8812f2eb_3b36_4677_a6a4_fccb4be01aba.slice/crio-c18a53e06f5b5991f462b8f74638de5bfab733069c3d02522585adba959c03ee WatchSource:0}: Error finding container c18a53e06f5b5991f462b8f74638de5bfab733069c3d02522585adba959c03ee: Status 404 returned error can't find the container with id c18a53e06f5b5991f462b8f74638de5bfab733069c3d02522585adba959c03ee
Jul 17 01:04:07.180982 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:07.180937    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" event={"ID":"8812f2eb-3b36-4677-a6a4-fccb4be01aba","Type":"ContainerStarted","Data":"df4c741fb39da8d03a7500726e0908ec6c82e2d4f350a8c18680bb82ddd48a74"}
Jul 17 01:04:07.181178 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:07.181007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" event={"ID":"8812f2eb-3b36-4677-a6a4-fccb4be01aba","Type":"ContainerStarted","Data":"c18a53e06f5b5991f462b8f74638de5bfab733069c3d02522585adba959c03ee"}
Jul 17 01:04:07.196761 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:07.196699    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" podStartSLOduration=1.19667813 podStartE2EDuration="1.19667813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:04:07.19390613 +0000 UTC m=+2841.770058707" watchObservedRunningTime="2026-07-17 01:04:07.19667813 +0000 UTC m=+2841.772830744"
Jul 17 01:04:07.962614 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:07.962509    2573 scope.go:117] "RemoveContainer" containerID="273ce39c9d6a35bc157264f5d5e6f1958505c65a31696c0b6679d389a6f5cf70"
Jul 17 01:04:08.189148 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:08.189112    2573 generic.go:356] "Generic (PLEG): container finished" podID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerID="df4c741fb39da8d03a7500726e0908ec6c82e2d4f350a8c18680bb82ddd48a74" exitCode=6
Jul 17 01:04:08.189371 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:08.189204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" event={"ID":"8812f2eb-3b36-4677-a6a4-fccb4be01aba","Type":"ContainerDied","Data":"df4c741fb39da8d03a7500726e0908ec6c82e2d4f350a8c18680bb82ddd48a74"}
Jul 17 01:04:08.189439 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:08.189407    2573 scope.go:117] "RemoveContainer" containerID="df4c741fb39da8d03a7500726e0908ec6c82e2d4f350a8c18680bb82ddd48a74"
Jul 17 01:04:09.197212 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.197104    2573 generic.go:356] "Generic (PLEG): container finished" podID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerID="d5ba19005cadd07a41cccf2e049b40adc47409fedaa032d404becb111aabb1dd" exitCode=6
Jul 17 01:04:09.197212 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.197152    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" event={"ID":"52ca902f-c507-473b-bb6b-17b4a42249c9","Type":"ContainerDied","Data":"d5ba19005cadd07a41cccf2e049b40adc47409fedaa032d404becb111aabb1dd"}
Jul 17 01:04:09.197893 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.197230    2573 scope.go:117] "RemoveContainer" containerID="273ce39c9d6a35bc157264f5d5e6f1958505c65a31696c0b6679d389a6f5cf70"
Jul 17 01:04:09.197893 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.197557    2573 scope.go:117] "RemoveContainer" containerID="d5ba19005cadd07a41cccf2e049b40adc47409fedaa032d404becb111aabb1dd"
Jul 17 01:04:09.197893 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:09.197875    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9_odh-ai-gateway-infra(52ca902f-c507-473b-bb6b-17b4a42249c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9"
Jul 17 01:04:09.199503 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.199473    2573 generic.go:356] "Generic (PLEG): container finished" podID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerID="3e8eda463f0a4192265c644a15fa791a0f52912bf8a7cce2aa7c1ba0a824ed68" exitCode=6
Jul 17 01:04:09.199619 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.199550    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" event={"ID":"8812f2eb-3b36-4677-a6a4-fccb4be01aba","Type":"ContainerDied","Data":"3e8eda463f0a4192265c644a15fa791a0f52912bf8a7cce2aa7c1ba0a824ed68"}
Jul 17 01:04:09.199771 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.199758    2573 scope.go:117] "RemoveContainer" containerID="3e8eda463f0a4192265c644a15fa791a0f52912bf8a7cce2aa7c1ba0a824ed68"
Jul 17 01:04:09.199993 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:09.199976    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-xtnbb_odh-ai-gateway-infra(8812f2eb-3b36-4677-a6a4-fccb4be01aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba"
Jul 17 01:04:09.209832 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:09.209766    2573 scope.go:117] "RemoveContainer" containerID="df4c741fb39da8d03a7500726e0908ec6c82e2d4f350a8c18680bb82ddd48a74"
Jul 17 01:04:10.208400 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:10.208361    2573 scope.go:117] "RemoveContainer" containerID="3e8eda463f0a4192265c644a15fa791a0f52912bf8a7cce2aa7c1ba0a824ed68"
Jul 17 01:04:10.208977 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:10.208635    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-xtnbb_odh-ai-gateway-infra(8812f2eb-3b36-4677-a6a4-fccb4be01aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba"
Jul 17 01:04:10.232046 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:10.231993    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"]
Jul 17 01:04:10.379534 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:10.379482    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"
Jul 17 01:04:10.510246 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:10.510190    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca902f-c507-473b-bb6b-17b4a42249c9-maas-api-service-ca\") pod \"52ca902f-c507-473b-bb6b-17b4a42249c9\" (UID: \"52ca902f-c507-473b-bb6b-17b4a42249c9\") "
Jul 17 01:04:10.510595 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:10.510570    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ca902f-c507-473b-bb6b-17b4a42249c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52ca902f-c507-473b-bb6b-17b4a42249c9" (UID: "52ca902f-c507-473b-bb6b-17b4a42249c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:04:10.611587 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:10.611536    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca902f-c507-473b-bb6b-17b4a42249c9-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:11.103764 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.103722    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"]
Jul 17 01:04:11.104090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.104056    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl" podUID="eeae9d43-2a24-4a1d-aebe-875c780d6e1a" containerName="maas-api" containerID="cri-o://b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f" gracePeriod=30
Jul 17 01:04:11.125050 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.124985    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl" podUID="eeae9d43-2a24-4a1d-aebe-875c780d6e1a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.137:8443/health\": dial tcp 10.133.0.137:8443: connect: connection refused"
Jul 17 01:04:11.217501 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.217465    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"
Jul 17 01:04:11.218086 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.217497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9" event={"ID":"52ca902f-c507-473b-bb6b-17b4a42249c9","Type":"ContainerDied","Data":"aff6f2fafce5dd19459039af3dda898f9355e3bb0a627e4118e43ac337eedd0c"}
Jul 17 01:04:11.218086 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.217581    2573 scope.go:117] "RemoveContainer" containerID="d5ba19005cadd07a41cccf2e049b40adc47409fedaa032d404becb111aabb1dd"
Jul 17 01:04:11.240224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.240183    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"]
Jul 17 01:04:11.247245 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.247181    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-4c2g9"]
Jul 17 01:04:11.387695 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.387652    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:04:11.519281 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.519231    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-maas-api-tls\") pod \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") "
Jul 17 01:04:11.519518 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.519318    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8jmx\" (UniqueName: \"kubernetes.io/projected/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-kube-api-access-h8jmx\") pod \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") "
Jul 17 01:04:11.519518 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.519348    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-openshift-service-ca\") pod \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\" (UID: \"eeae9d43-2a24-4a1d-aebe-875c780d6e1a\") "
Jul 17 01:04:11.519854 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.519780    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eeae9d43-2a24-4a1d-aebe-875c780d6e1a" (UID: "eeae9d43-2a24-4a1d-aebe-875c780d6e1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:04:11.521739 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.521705    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eeae9d43-2a24-4a1d-aebe-875c780d6e1a" (UID: "eeae9d43-2a24-4a1d-aebe-875c780d6e1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 01:04:11.521911 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.521853    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-kube-api-access-h8jmx" (OuterVolumeSpecName: "kube-api-access-h8jmx") pod "eeae9d43-2a24-4a1d-aebe-875c780d6e1a" (UID: "eeae9d43-2a24-4a1d-aebe-875c780d6e1a"). InnerVolumeSpecName "kube-api-access-h8jmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:04:11.620463 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.620355    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:11.620463 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.620395    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8jmx\" (UniqueName: \"kubernetes.io/projected/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-kube-api-access-h8jmx\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:11.620463 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.620405    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeae9d43-2a24-4a1d-aebe-875c780d6e1a-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:11.969295 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:11.969186    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" path="/var/lib/kubelet/pods/52ca902f-c507-473b-bb6b-17b4a42249c9/volumes"
Jul 17 01:04:12.226025 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.225907    2573 generic.go:356] "Generic (PLEG): container finished" podID="eeae9d43-2a24-4a1d-aebe-875c780d6e1a" containerID="b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f" exitCode=0
Jul 17 01:04:12.226025 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.225961    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"
Jul 17 01:04:12.226025 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.225989    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl" event={"ID":"eeae9d43-2a24-4a1d-aebe-875c780d6e1a","Type":"ContainerDied","Data":"b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f"}
Jul 17 01:04:12.226679 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.226059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl" event={"ID":"eeae9d43-2a24-4a1d-aebe-875c780d6e1a","Type":"ContainerDied","Data":"1e4f58054c029e4c3549d3a4e908235b1b59189f5183c11bb0186d57a805e322"}
Jul 17 01:04:12.226679 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.226084    2573 scope.go:117] "RemoveContainer" containerID="b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f"
Jul 17 01:04:12.236558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.236534    2573 scope.go:117] "RemoveContainer" containerID="b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f"
Jul 17 01:04:12.236978 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:12.236949    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f\": container with ID starting with b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f not found: ID does not exist" containerID="b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f"
Jul 17 01:04:12.237066 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.236993    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f"} err="failed to get container status \"b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f\": rpc error: code = NotFound desc = could not find container \"b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f\": container with ID starting with b354ab6464bbe8d53af36eb37374eeeefeab46cf5105b233b9a7844a7fbd569f not found: ID does not exist"
Jul 17 01:04:12.243646 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.243602    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"]
Jul 17 01:04:12.247602 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:12.247558    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1e207-7f5cc87c5b-rfhrl"]
Jul 17 01:04:13.962639 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:13.962607    2573 scope.go:117] "RemoveContainer" containerID="ff360fb627e0f8396cdf19daa91a36b24051b81b5c44af5dedf5eec321f5fb96"
Jul 17 01:04:13.968702 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:13.968671    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeae9d43-2a24-4a1d-aebe-875c780d6e1a" path="/var/lib/kubelet/pods/eeae9d43-2a24-4a1d-aebe-875c780d6e1a/volumes"
Jul 17 01:04:15.249725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:15.249676    2573 generic.go:356] "Generic (PLEG): container finished" podID="b9bf630a-14a1-412c-8e41-0298e727f002" containerID="0187a21913d27f203a14a3df731be1e75814c60d303d624b70f5a2c28a39b717" exitCode=6
Jul 17 01:04:15.250274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:15.249760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" event={"ID":"b9bf630a-14a1-412c-8e41-0298e727f002","Type":"ContainerDied","Data":"0187a21913d27f203a14a3df731be1e75814c60d303d624b70f5a2c28a39b717"}
Jul 17 01:04:15.250274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:15.249941    2573 scope.go:117] "RemoveContainer" containerID="ff360fb627e0f8396cdf19daa91a36b24051b81b5c44af5dedf5eec321f5fb96"
Jul 17 01:04:15.250421 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:15.250398    2573 scope.go:117] "RemoveContainer" containerID="0187a21913d27f203a14a3df731be1e75814c60d303d624b70f5a2c28a39b717"
Jul 17 01:04:15.250756 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:15.250727    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq_odh-ai-gateway-infra(b9bf630a-14a1-412c-8e41-0298e727f002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" podUID="b9bf630a-14a1-412c-8e41-0298e727f002"
Jul 17 01:04:16.273771 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:16.273728    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"]
Jul 17 01:04:16.419711 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:16.419681    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"
Jul 17 01:04:16.568978 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:16.568935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bf630a-14a1-412c-8e41-0298e727f002-maas-api-service-ca\") pod \"b9bf630a-14a1-412c-8e41-0298e727f002\" (UID: \"b9bf630a-14a1-412c-8e41-0298e727f002\") "
Jul 17 01:04:16.569300 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:16.569276    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bf630a-14a1-412c-8e41-0298e727f002-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9bf630a-14a1-412c-8e41-0298e727f002" (UID: "b9bf630a-14a1-412c-8e41-0298e727f002"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:04:16.670258 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:16.670158    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bf630a-14a1-412c-8e41-0298e727f002-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:17.268159 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:17.268124    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"
Jul 17 01:04:17.268384 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:17.268158    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq" event={"ID":"b9bf630a-14a1-412c-8e41-0298e727f002","Type":"ContainerDied","Data":"bdd5facdc3bb873748b1240e45b8bb915f3651412957a545b566b0dd4bc5d8e3"}
Jul 17 01:04:17.268384 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:17.268234    2573 scope.go:117] "RemoveContainer" containerID="0187a21913d27f203a14a3df731be1e75814c60d303d624b70f5a2c28a39b717"
Jul 17 01:04:17.290713 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:17.290665    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"]
Jul 17 01:04:17.294246 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:17.294210    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-hm7pq"]
Jul 17 01:04:17.969763 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:17.969715    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" path="/var/lib/kubelet/pods/b9bf630a-14a1-412c-8e41-0298e727f002/volumes"
Jul 17 01:04:20.963056 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:20.963022    2573 scope.go:117] "RemoveContainer" containerID="f105918b04aa8df64ddea0705894a32cb936abbe2d71f75f8a64d04002b0499f"
Jul 17 01:04:21.294725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:21.294687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" event={"ID":"f8836e04-b365-439f-9057-4493d2ebf5df","Type":"ContainerStarted","Data":"0035c756dc5ed0912fe9fe16a6e736dcbfecca9195f83e73bf77ac73505ff56c"}
Jul 17 01:04:22.306413 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:22.306372    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8836e04-b365-439f-9057-4493d2ebf5df" containerID="0035c756dc5ed0912fe9fe16a6e736dcbfecca9195f83e73bf77ac73505ff56c" exitCode=6
Jul 17 01:04:22.306955 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:22.306444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" event={"ID":"f8836e04-b365-439f-9057-4493d2ebf5df","Type":"ContainerDied","Data":"0035c756dc5ed0912fe9fe16a6e736dcbfecca9195f83e73bf77ac73505ff56c"}
Jul 17 01:04:22.306955 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:22.306519    2573 scope.go:117] "RemoveContainer" containerID="f105918b04aa8df64ddea0705894a32cb936abbe2d71f75f8a64d04002b0499f"
Jul 17 01:04:22.306955 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:22.306869    2573 scope.go:117] "RemoveContainer" containerID="0035c756dc5ed0912fe9fe16a6e736dcbfecca9195f83e73bf77ac73505ff56c"
Jul 17 01:04:22.307189 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:22.307169    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99_odh-ai-gateway-infra(f8836e04-b365-439f-9057-4493d2ebf5df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" podUID="f8836e04-b365-439f-9057-4493d2ebf5df"
Jul 17 01:04:22.321454 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:22.321421    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"]
Jul 17 01:04:23.484420 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:23.484383    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"
Jul 17 01:04:23.641476 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:23.641366    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8836e04-b365-439f-9057-4493d2ebf5df-maas-api-service-ca\") pod \"f8836e04-b365-439f-9057-4493d2ebf5df\" (UID: \"f8836e04-b365-439f-9057-4493d2ebf5df\") "
Jul 17 01:04:23.641768 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:23.641742    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8836e04-b365-439f-9057-4493d2ebf5df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8836e04-b365-439f-9057-4493d2ebf5df" (UID: "f8836e04-b365-439f-9057-4493d2ebf5df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:04:23.742239 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:23.742199    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8836e04-b365-439f-9057-4493d2ebf5df-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:23.962451 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:23.962336    2573 scope.go:117] "RemoveContainer" containerID="3e8eda463f0a4192265c644a15fa791a0f52912bf8a7cce2aa7c1ba0a824ed68"
Jul 17 01:04:24.322153 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:24.322102    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"
Jul 17 01:04:24.322153 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:24.322133    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99" event={"ID":"f8836e04-b365-439f-9057-4493d2ebf5df","Type":"ContainerDied","Data":"3bfaa5407e9d13bc73b50090a193c5e130b20a6b62df4cf148c32259965f490c"}
Jul 17 01:04:24.322445 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:24.322260    2573 scope.go:117] "RemoveContainer" containerID="0035c756dc5ed0912fe9fe16a6e736dcbfecca9195f83e73bf77ac73505ff56c"
Jul 17 01:04:24.324751 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:24.324712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" event={"ID":"8812f2eb-3b36-4677-a6a4-fccb4be01aba","Type":"ContainerStarted","Data":"5ab6f17514110e0867cea9cbad603efeaef5d6b29988caa134626fa501a27772"}
Jul 17 01:04:24.351283 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:24.351241    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"]
Jul 17 01:04:24.355091 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:24.355054    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9sw99"]
Jul 17 01:04:25.333646 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:25.333606    2573 generic.go:356] "Generic (PLEG): container finished" podID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerID="5ab6f17514110e0867cea9cbad603efeaef5d6b29988caa134626fa501a27772" exitCode=6
Jul 17 01:04:25.334188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:25.333677    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" event={"ID":"8812f2eb-3b36-4677-a6a4-fccb4be01aba","Type":"ContainerDied","Data":"5ab6f17514110e0867cea9cbad603efeaef5d6b29988caa134626fa501a27772"}
Jul 17 01:04:25.334188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:25.333740    2573 scope.go:117] "RemoveContainer" containerID="3e8eda463f0a4192265c644a15fa791a0f52912bf8a7cce2aa7c1ba0a824ed68"
Jul 17 01:04:25.334188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:25.334112    2573 scope.go:117] "RemoveContainer" containerID="5ab6f17514110e0867cea9cbad603efeaef5d6b29988caa134626fa501a27772"
Jul 17 01:04:25.334352 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:25.334335    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-xtnbb_odh-ai-gateway-infra(8812f2eb-3b36-4677-a6a4-fccb4be01aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba"
Jul 17 01:04:25.348551 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:25.348518    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"]
Jul 17 01:04:25.968879 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:25.968844    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" path="/var/lib/kubelet/pods/f8836e04-b365-439f-9057-4493d2ebf5df/volumes"
Jul 17 01:04:26.516291 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:26.516249    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"
Jul 17 01:04:26.672644 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:26.672526    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8812f2eb-3b36-4677-a6a4-fccb4be01aba-maas-api-service-ca\") pod \"8812f2eb-3b36-4677-a6a4-fccb4be01aba\" (UID: \"8812f2eb-3b36-4677-a6a4-fccb4be01aba\") "
Jul 17 01:04:26.673018 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:26.672981    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8812f2eb-3b36-4677-a6a4-fccb4be01aba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8812f2eb-3b36-4677-a6a4-fccb4be01aba" (UID: "8812f2eb-3b36-4677-a6a4-fccb4be01aba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:04:26.774369 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:26.774310    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8812f2eb-3b36-4677-a6a4-fccb4be01aba-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:04:27.351441 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:27.351408    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"
Jul 17 01:04:27.351691 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:27.351406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb" event={"ID":"8812f2eb-3b36-4677-a6a4-fccb4be01aba","Type":"ContainerDied","Data":"c18a53e06f5b5991f462b8f74638de5bfab733069c3d02522585adba959c03ee"}
Jul 17 01:04:27.351691 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:27.351555    2573 scope.go:117] "RemoveContainer" containerID="5ab6f17514110e0867cea9cbad603efeaef5d6b29988caa134626fa501a27772"
Jul 17 01:04:27.376927 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:27.376881    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"]
Jul 17 01:04:27.382110 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:27.382071    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-xtnbb"]
Jul 17 01:04:27.969922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:27.969879    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" path="/var/lib/kubelet/pods/8812f2eb-3b36-4677-a6a4-fccb4be01aba/volumes"
Jul 17 01:04:30.000366 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000318    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"]
Jul 17 01:04:30.000965 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000923    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:30.000965 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000943    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:30.000965 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000956    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.000965 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000965    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000976    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000985    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.000995    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001002    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001016    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001024    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001034    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001042    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001052    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001061    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001089    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeae9d43-2a24-4a1d-aebe-875c780d6e1a" containerName="maas-api"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001097    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeae9d43-2a24-4a1d-aebe-875c780d6e1a" containerName="maas-api"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001113    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001124    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001140    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:30.001196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001149    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001233    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001245    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001254    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001262    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001272    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeae9d43-2a24-4a1d-aebe-875c780d6e1a" containerName="maas-api"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001282    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001290    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001298    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001305    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001314    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001321    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:30.002008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.001330    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9bf630a-14a1-412c-8e41-0298e727f002" containerName="revoke-keys"
Jul 17 01:04:30.006146 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.006114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.008390 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.008356    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b5ab5c-serving-cert\""
Jul 17 01:04:30.009367 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.009335    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 01:04:30.013454 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.013420    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"]
Jul 17 01:04:30.109173 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.109130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4pn\" (UniqueName: \"kubernetes.io/projected/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-kube-api-access-qv4pn\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.109456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.109415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.109668 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.109638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.211191 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.211142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.211420 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.211214    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.211420 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.211256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4pn\" (UniqueName: \"kubernetes.io/projected/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-kube-api-access-qv4pn\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.211420 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:30.211308    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-serving-cert: secret "maas-api-e2e-api-a-b5ab5c-serving-cert" not found
Jul 17 01:04:30.211420 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:30.211377    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls podName:1aaf5dc6-d42d-41da-9a47-a126c35b06e2 nodeName:}" failed. No retries permitted until 2026-07-17 01:04:30.711358918 +0000 UTC m=+2865.287511470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls") pod "maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl" (UID: "1aaf5dc6-d42d-41da-9a47-a126c35b06e2") : secret "maas-api-e2e-api-a-b5ab5c-serving-cert" not found
Jul 17 01:04:30.211945 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.211927    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.226972 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.226930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4pn\" (UniqueName: \"kubernetes.io/projected/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-kube-api-access-qv4pn\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.716635 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.716575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.719740 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.719705    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls\") pod \"maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:30.921154 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:30.921108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:31.078171 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:31.078144    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"]
Jul 17 01:04:31.080341 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:04:31.080307    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaf5dc6_d42d_41da_9a47_a126c35b06e2.slice/crio-bc884bdd8d6cc8824ecefc3c25db6c94c8afd3d349beb153fc29f1ebdfdcd95e WatchSource:0}: Error finding container bc884bdd8d6cc8824ecefc3c25db6c94c8afd3d349beb153fc29f1ebdfdcd95e: Status 404 returned error can't find the container with id bc884bdd8d6cc8824ecefc3c25db6c94c8afd3d349beb153fc29f1ebdfdcd95e
Jul 17 01:04:31.081816 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:31.081767    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 01:04:31.380174 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:31.380136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl" event={"ID":"1aaf5dc6-d42d-41da-9a47-a126c35b06e2","Type":"ContainerStarted","Data":"bc884bdd8d6cc8824ecefc3c25db6c94c8afd3d349beb153fc29f1ebdfdcd95e"}
Jul 17 01:04:32.395398 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:32.395362    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl" event={"ID":"1aaf5dc6-d42d-41da-9a47-a126c35b06e2","Type":"ContainerStarted","Data":"49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a"}
Jul 17 01:04:32.395904 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:32.395459    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:32.412342 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:32.412276    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl" podStartSLOduration=2.74241674 podStartE2EDuration="3.412254419s" totalImagesPullingTime="669.837679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:04:32.409570611 +0000 UTC m=+2866.985723187" watchObservedRunningTime="2026-07-17 01:04:32.412254419 +0000 UTC m=+2866.988407035"
Jul 17 01:04:37.988013 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.987736    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"]
Jul 17 01:04:37.989090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.989005    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:37.989090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.989045    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8836e04-b365-439f-9057-4493d2ebf5df" containerName="revoke-keys"
Jul 17 01:04:37.989090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.989059    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:37.989090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.989071    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:37.989348 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.989124    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:37.989348 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.989135    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ca902f-c507-473b-bb6b-17b4a42249c9" containerName="revoke-keys"
Jul 17 01:04:37.989348 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.989247    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8812f2eb-3b36-4677-a6a4-fccb4be01aba" containerName="revoke-keys"
Jul 17 01:04:37.992678 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.992647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:37.994542 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:37.994518    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-49fd04-serving-cert\""
Jul 17 01:04:38.003290 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.003252    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"]
Jul 17 01:04:38.085810 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.085755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d7f187-cf98-48bb-83a2-19b378328312-openshift-service-ca\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.086034 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.085855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.086034 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.085910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c6zm\" (UniqueName: \"kubernetes.io/projected/77d7f187-cf98-48bb-83a2-19b378328312-kube-api-access-9c6zm\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.186656 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.186585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c6zm\" (UniqueName: \"kubernetes.io/projected/77d7f187-cf98-48bb-83a2-19b378328312-kube-api-access-9c6zm\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.186948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.186692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d7f187-cf98-48bb-83a2-19b378328312-openshift-service-ca\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.186948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.186736    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.186948 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:38.186870    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-serving-cert: secret "maas-api-e2e-api-b-49fd04-serving-cert" not found
Jul 17 01:04:38.186948 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:38.186938    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls podName:77d7f187-cf98-48bb-83a2-19b378328312 nodeName:}" failed. No retries permitted until 2026-07-17 01:04:38.686916233 +0000 UTC m=+2873.263068790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls") pod "maas-api-e2e-api-b-49fd04-75974997b4-s8nch" (UID: "77d7f187-cf98-48bb-83a2-19b378328312") : secret "maas-api-e2e-api-b-49fd04-serving-cert" not found
Jul 17 01:04:38.187428 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.187407    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d7f187-cf98-48bb-83a2-19b378328312-openshift-service-ca\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.195435 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.195394    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c6zm\" (UniqueName: \"kubernetes.io/projected/77d7f187-cf98-48bb-83a2-19b378328312-kube-api-access-9c6zm\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.408113 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.408046    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:04:38.693828 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.693655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.696566 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.696527    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls\") pod \"maas-api-e2e-api-b-49fd04-75974997b4-s8nch\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:38.906292 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:38.906242    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:39.064079 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:39.064005    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"]
Jul 17 01:04:39.067066 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:04:39.067029    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d7f187_cf98_48bb_83a2_19b378328312.slice/crio-2606a9d0ac3b9ed7c4523366a4c183b307b085b16a99d202bb377995d660f18c WatchSource:0}: Error finding container 2606a9d0ac3b9ed7c4523366a4c183b307b085b16a99d202bb377995d660f18c: Status 404 returned error can't find the container with id 2606a9d0ac3b9ed7c4523366a4c183b307b085b16a99d202bb377995d660f18c
Jul 17 01:04:39.441162 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:39.441120    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch" event={"ID":"77d7f187-cf98-48bb-83a2-19b378328312","Type":"ContainerStarted","Data":"2606a9d0ac3b9ed7c4523366a4c183b307b085b16a99d202bb377995d660f18c"}
Jul 17 01:04:40.450555 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:40.450510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch" event={"ID":"77d7f187-cf98-48bb-83a2-19b378328312","Type":"ContainerStarted","Data":"1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b"}
Jul 17 01:04:40.451224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:40.450643    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:40.467450 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:40.467375    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch" podStartSLOduration=3.080533337 podStartE2EDuration="3.467348165s" totalImagesPullingTime="386.814828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:04:40.464640706 +0000 UTC m=+2875.040793279" watchObservedRunningTime="2026-07-17 01:04:40.467348165 +0000 UTC m=+2875.043500738"
Jul 17 01:04:46.463342 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:46.463303    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:04:49.111003 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.110959    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"]
Jul 17 01:04:49.114882 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.114856    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"
Jul 17 01:04:49.116774 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.116739    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:04:49.121851 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.121816    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"]
Jul 17 01:04:49.198690 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.198645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s\" (UID: \"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"
Jul 17 01:04:49.299580 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.299511    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s\" (UID: \"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"
Jul 17 01:04:49.300233 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.300209    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s\" (UID: \"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"
Jul 17 01:04:49.430191 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.430074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"
Jul 17 01:04:49.587685 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:49.587652    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"]
Jul 17 01:04:49.589614 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:04:49.589583    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb859d2a7_ffc5_4434_a3e7_0a98f54a9e9a.slice/crio-57f2372bff56794b98ecb898edafbed86d9a540fc66a9b38bb8571cd728c0d1c WatchSource:0}: Error finding container 57f2372bff56794b98ecb898edafbed86d9a540fc66a9b38bb8571cd728c0d1c: Status 404 returned error can't find the container with id 57f2372bff56794b98ecb898edafbed86d9a540fc66a9b38bb8571cd728c0d1c
Jul 17 01:04:50.524784 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:50.524739    2573 generic.go:356] "Generic (PLEG): container finished" podID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerID="7b8a730630039c458fe048948473beaa96e4cd7e81c04844797bec15ed3114eb" exitCode=6
Jul 17 01:04:50.525274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:50.524827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" event={"ID":"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a","Type":"ContainerDied","Data":"7b8a730630039c458fe048948473beaa96e4cd7e81c04844797bec15ed3114eb"}
Jul 17 01:04:50.525274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:50.524923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" event={"ID":"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a","Type":"ContainerStarted","Data":"57f2372bff56794b98ecb898edafbed86d9a540fc66a9b38bb8571cd728c0d1c"}
Jul 17 01:04:50.525274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:50.525148    2573 scope.go:117] "RemoveContainer" containerID="7b8a730630039c458fe048948473beaa96e4cd7e81c04844797bec15ed3114eb"
Jul 17 01:04:51.533009 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:51.532952    2573 generic.go:356] "Generic (PLEG): container finished" podID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerID="20e9c25678c5ba1d27b0189157df6e83471ab4c73c05197a09ed6e55861bcc83" exitCode=6
Jul 17 01:04:51.533560 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:51.533039    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" event={"ID":"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a","Type":"ContainerDied","Data":"20e9c25678c5ba1d27b0189157df6e83471ab4c73c05197a09ed6e55861bcc83"}
Jul 17 01:04:51.533560 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:51.533121    2573 scope.go:117] "RemoveContainer" containerID="7b8a730630039c458fe048948473beaa96e4cd7e81c04844797bec15ed3114eb"
Jul 17 01:04:51.533560 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:51.533395    2573 scope.go:117] "RemoveContainer" containerID="20e9c25678c5ba1d27b0189157df6e83471ab4c73c05197a09ed6e55861bcc83"
Jul 17 01:04:51.533702 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:51.533669    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s_odh-ai-gateway-infra(b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a"
Jul 17 01:04:52.541303 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:04:52.541271    2573 scope.go:117] "RemoveContainer" containerID="20e9c25678c5ba1d27b0189157df6e83471ab4c73c05197a09ed6e55861bcc83"
Jul 17 01:04:52.541903 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:04:52.541484    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s_odh-ai-gateway-infra(b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a"
Jul 17 01:05:01.039216 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.038527    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"]
Jul 17 01:05:01.039810 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.039261    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl" podUID="1aaf5dc6-d42d-41da-9a47-a126c35b06e2" containerName="maas-api" containerID="cri-o://49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a" gracePeriod=30
Jul 17 01:05:01.316504 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.316454    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:05:01.422782 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.422741    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv4pn\" (UniqueName: \"kubernetes.io/projected/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-kube-api-access-qv4pn\") pod \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") "
Jul 17 01:05:01.423008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.422892    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls\") pod \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") "
Jul 17 01:05:01.423008 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.422929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-openshift-service-ca\") pod \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\" (UID: \"1aaf5dc6-d42d-41da-9a47-a126c35b06e2\") "
Jul 17 01:05:01.423363 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.423334    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1aaf5dc6-d42d-41da-9a47-a126c35b06e2" (UID: "1aaf5dc6-d42d-41da-9a47-a126c35b06e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:05:01.425226 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.425190    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1aaf5dc6-d42d-41da-9a47-a126c35b06e2" (UID: "1aaf5dc6-d42d-41da-9a47-a126c35b06e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 01:05:01.425226 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.425205    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-kube-api-access-qv4pn" (OuterVolumeSpecName: "kube-api-access-qv4pn") pod "1aaf5dc6-d42d-41da-9a47-a126c35b06e2" (UID: "1aaf5dc6-d42d-41da-9a47-a126c35b06e2"). InnerVolumeSpecName "kube-api-access-qv4pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:05:01.523999 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.523955    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:01.523999 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.523990    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:01.523999 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.524001    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv4pn\" (UniqueName: \"kubernetes.io/projected/1aaf5dc6-d42d-41da-9a47-a126c35b06e2-kube-api-access-qv4pn\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:01.601304 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.601200    2573 generic.go:356] "Generic (PLEG): container finished" podID="1aaf5dc6-d42d-41da-9a47-a126c35b06e2" containerID="49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a" exitCode=0
Jul 17 01:05:01.601304 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.601249    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"
Jul 17 01:05:01.601304 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.601289    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl" event={"ID":"1aaf5dc6-d42d-41da-9a47-a126c35b06e2","Type":"ContainerDied","Data":"49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a"}
Jul 17 01:05:01.601586 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.601350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl" event={"ID":"1aaf5dc6-d42d-41da-9a47-a126c35b06e2","Type":"ContainerDied","Data":"bc884bdd8d6cc8824ecefc3c25db6c94c8afd3d349beb153fc29f1ebdfdcd95e"}
Jul 17 01:05:01.601586 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.601377    2573 scope.go:117] "RemoveContainer" containerID="49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a"
Jul 17 01:05:01.615919 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.615207    2573 scope.go:117] "RemoveContainer" containerID="49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a"
Jul 17 01:05:01.615919 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:01.615687    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a\": container with ID starting with 49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a not found: ID does not exist" containerID="49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a"
Jul 17 01:05:01.615919 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.615746    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a"} err="failed to get container status \"49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a\": rpc error: code = NotFound desc = could not find container \"49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a\": container with ID starting with 49545a84dc3a75b5076c508db86e65d0d422697e452cd366b96f4b354f6d657a not found: ID does not exist"
Jul 17 01:05:01.629734 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.629687    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"]
Jul 17 01:05:01.635338 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.635298    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b5ab5c-7d47ff4b96-5h9gl"]
Jul 17 01:05:01.968889 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:01.968768    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aaf5dc6-d42d-41da-9a47-a126c35b06e2" path="/var/lib/kubelet/pods/1aaf5dc6-d42d-41da-9a47-a126c35b06e2/volumes"
Jul 17 01:05:06.442399 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.442357    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"]
Jul 17 01:05:06.442879 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.442814    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aaf5dc6-d42d-41da-9a47-a126c35b06e2" containerName="maas-api"
Jul 17 01:05:06.442879 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.442833    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aaf5dc6-d42d-41da-9a47-a126c35b06e2" containerName="maas-api"
Jul 17 01:05:06.442965 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.442896    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aaf5dc6-d42d-41da-9a47-a126c35b06e2" containerName="maas-api"
Jul 17 01:05:06.445948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.445921    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"
Jul 17 01:05:06.464743 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.464698    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"]
Jul 17 01:05:06.569511 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.569462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b572604b-0c21-42f9-9734-edd55753f99c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-vbxj5\" (UID: \"b572604b-0c21-42f9-9734-edd55753f99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"
Jul 17 01:05:06.671006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.670955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b572604b-0c21-42f9-9734-edd55753f99c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-vbxj5\" (UID: \"b572604b-0c21-42f9-9734-edd55753f99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"
Jul 17 01:05:06.671641 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.671619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b572604b-0c21-42f9-9734-edd55753f99c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-vbxj5\" (UID: \"b572604b-0c21-42f9-9734-edd55753f99c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"
Jul 17 01:05:06.757727 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.757687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"
Jul 17 01:05:06.914386 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:06.914339    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"]
Jul 17 01:05:06.915497 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:05:06.915461    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb572604b_0c21_42f9_9734_edd55753f99c.slice/crio-a7ed2b9ae0a337cb23828f71ad92564ef7b57db3bff0139d2cc4376b4622c71f WatchSource:0}: Error finding container a7ed2b9ae0a337cb23828f71ad92564ef7b57db3bff0139d2cc4376b4622c71f: Status 404 returned error can't find the container with id a7ed2b9ae0a337cb23828f71ad92564ef7b57db3bff0139d2cc4376b4622c71f
Jul 17 01:05:07.644734 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:07.644679    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" event={"ID":"b572604b-0c21-42f9-9734-edd55753f99c","Type":"ContainerStarted","Data":"b054970e81882572fe31d399e3ca31468a8af86d518f328cb0bb7b147eb2134a"}
Jul 17 01:05:07.645273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:07.644758    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" event={"ID":"b572604b-0c21-42f9-9734-edd55753f99c","Type":"ContainerStarted","Data":"a7ed2b9ae0a337cb23828f71ad92564ef7b57db3bff0139d2cc4376b4622c71f"}
Jul 17 01:05:07.668706 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:07.665722    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" podStartSLOduration=1.665690193 podStartE2EDuration="1.665690193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:05:07.658432495 +0000 UTC m=+2902.234585072" watchObservedRunningTime="2026-07-17 01:05:07.665690193 +0000 UTC m=+2902.241842769"
Jul 17 01:05:07.963647 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:07.963530    2573 scope.go:117] "RemoveContainer" containerID="20e9c25678c5ba1d27b0189157df6e83471ab4c73c05197a09ed6e55861bcc83"
Jul 17 01:05:08.654834 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:08.654751    2573 generic.go:356] "Generic (PLEG): container finished" podID="b572604b-0c21-42f9-9734-edd55753f99c" containerID="b054970e81882572fe31d399e3ca31468a8af86d518f328cb0bb7b147eb2134a" exitCode=6
Jul 17 01:05:08.654834 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:08.654828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" event={"ID":"b572604b-0c21-42f9-9734-edd55753f99c","Type":"ContainerDied","Data":"b054970e81882572fe31d399e3ca31468a8af86d518f328cb0bb7b147eb2134a"}
Jul 17 01:05:08.655389 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:08.655145    2573 scope.go:117] "RemoveContainer" containerID="b054970e81882572fe31d399e3ca31468a8af86d518f328cb0bb7b147eb2134a"
Jul 17 01:05:08.657210 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:08.657139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" event={"ID":"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a","Type":"ContainerStarted","Data":"58b8bafe7faa4197d185266b2e830a06eb35c07efc3bf82b5c3eaaf682bce74e"}
Jul 17 01:05:08.683882 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:08.683821    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" podStartSLOduration=19.683773035 podStartE2EDuration="19.683773035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:05:08.680763902 +0000 UTC m=+2903.256916478" watchObservedRunningTime="2026-07-17 01:05:08.683773035 +0000 UTC m=+2903.259925608"
Jul 17 01:05:09.666499 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.666382    2573 generic.go:356] "Generic (PLEG): container finished" podID="b572604b-0c21-42f9-9734-edd55753f99c" containerID="b14e358bfb0d6c1f41c3ac03ac6d06f009a92e8361c2618411ab512552a98192" exitCode=6
Jul 17 01:05:09.667027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.666482    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" event={"ID":"b572604b-0c21-42f9-9734-edd55753f99c","Type":"ContainerDied","Data":"b14e358bfb0d6c1f41c3ac03ac6d06f009a92e8361c2618411ab512552a98192"}
Jul 17 01:05:09.667027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.666571    2573 scope.go:117] "RemoveContainer" containerID="b054970e81882572fe31d399e3ca31468a8af86d518f328cb0bb7b147eb2134a"
Jul 17 01:05:09.667027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.666890    2573 scope.go:117] "RemoveContainer" containerID="b14e358bfb0d6c1f41c3ac03ac6d06f009a92e8361c2618411ab512552a98192"
Jul 17 01:05:09.667201 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:09.667175    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-vbxj5_odh-ai-gateway-infra(b572604b-0c21-42f9-9734-edd55753f99c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" podUID="b572604b-0c21-42f9-9734-edd55753f99c"
Jul 17 01:05:09.668658 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.668634    2573 generic.go:356] "Generic (PLEG): container finished" podID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerID="58b8bafe7faa4197d185266b2e830a06eb35c07efc3bf82b5c3eaaf682bce74e" exitCode=6
Jul 17 01:05:09.668828 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.668713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" event={"ID":"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a","Type":"ContainerDied","Data":"58b8bafe7faa4197d185266b2e830a06eb35c07efc3bf82b5c3eaaf682bce74e"}
Jul 17 01:05:09.669082 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.669065    2573 scope.go:117] "RemoveContainer" containerID="58b8bafe7faa4197d185266b2e830a06eb35c07efc3bf82b5c3eaaf682bce74e"
Jul 17 01:05:09.669287 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:09.669267    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s_odh-ai-gateway-infra(b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a"
Jul 17 01:05:09.680168 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.680147    2573 scope.go:117] "RemoveContainer" containerID="20e9c25678c5ba1d27b0189157df6e83471ab4c73c05197a09ed6e55861bcc83"
Jul 17 01:05:09.690380 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:09.690330    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"]
Jul 17 01:05:10.677115 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:10.677081    2573 scope.go:117] "RemoveContainer" containerID="b14e358bfb0d6c1f41c3ac03ac6d06f009a92e8361c2618411ab512552a98192"
Jul 17 01:05:10.677623 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:10.677324    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-vbxj5_odh-ai-gateway-infra(b572604b-0c21-42f9-9734-edd55753f99c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" podUID="b572604b-0c21-42f9-9734-edd55753f99c"
Jul 17 01:05:10.844488 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:10.844415    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"
Jul 17 01:05:11.014942 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.014904    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a-maas-api-service-ca\") pod \"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a\" (UID: \"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a\") "
Jul 17 01:05:11.015294 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.015270    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" (UID: "b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:05:11.116153 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.116092    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:11.686959 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.686926    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"
Jul 17 01:05:11.687558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.686925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s" event={"ID":"b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a","Type":"ContainerDied","Data":"57f2372bff56794b98ecb898edafbed86d9a540fc66a9b38bb8571cd728c0d1c"}
Jul 17 01:05:11.687558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.687075    2573 scope.go:117] "RemoveContainer" containerID="58b8bafe7faa4197d185266b2e830a06eb35c07efc3bf82b5c3eaaf682bce74e"
Jul 17 01:05:11.712358 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.712289    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"]
Jul 17 01:05:11.715221 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.715184    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bdl5s"]
Jul 17 01:05:11.969489 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:11.969381    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" path="/var/lib/kubelet/pods/b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a/volumes"
Jul 17 01:05:16.298076 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298032    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"]
Jul 17 01:05:16.298603 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298583    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.298652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298606    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.298652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298621    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.298652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298630    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.298652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298643    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.298652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298649    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.298846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298722    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.298846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.298731    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:16.303416 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.303381    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"
Jul 17 01:05:16.308865 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.308817    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"]
Jul 17 01:05:16.362263 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.362226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115ab490-eada-4fd1-9214-b204629c67d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4\" (UID: \"115ab490-eada-4fd1-9214-b204629c67d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"
Jul 17 01:05:16.462900 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.462756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115ab490-eada-4fd1-9214-b204629c67d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4\" (UID: \"115ab490-eada-4fd1-9214-b204629c67d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"
Jul 17 01:05:16.463591 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.463560    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115ab490-eada-4fd1-9214-b204629c67d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4\" (UID: \"115ab490-eada-4fd1-9214-b204629c67d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"
Jul 17 01:05:16.618274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.618235    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"
Jul 17 01:05:16.785515 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:16.785488    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"]
Jul 17 01:05:16.788031 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:05:16.787991    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115ab490_eada_4fd1_9214_b204629c67d3.slice/crio-78da401d11bd461145eccf45dd692f30d4f20e48510192ec3146baf41f700e3d WatchSource:0}: Error finding container 78da401d11bd461145eccf45dd692f30d4f20e48510192ec3146baf41f700e3d: Status 404 returned error can't find the container with id 78da401d11bd461145eccf45dd692f30d4f20e48510192ec3146baf41f700e3d
Jul 17 01:05:17.734963 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:17.734920    2573 generic.go:356] "Generic (PLEG): container finished" podID="115ab490-eada-4fd1-9214-b204629c67d3" containerID="5eb8f79168453a4d09722d3a7166273915d134a309ab3854c21e39bcf83458fa" exitCode=0
Jul 17 01:05:17.735410 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:17.735009    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4" event={"ID":"115ab490-eada-4fd1-9214-b204629c67d3","Type":"ContainerDied","Data":"5eb8f79168453a4d09722d3a7166273915d134a309ab3854c21e39bcf83458fa"}
Jul 17 01:05:17.735410 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:17.735110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4" event={"ID":"115ab490-eada-4fd1-9214-b204629c67d3","Type":"ContainerStarted","Data":"78da401d11bd461145eccf45dd692f30d4f20e48510192ec3146baf41f700e3d"}
Jul 17 01:05:18.895802 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:18.895749    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"
Jul 17 01:05:19.080619 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:19.080578    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115ab490-eada-4fd1-9214-b204629c67d3-maas-api-service-ca\") pod \"115ab490-eada-4fd1-9214-b204629c67d3\" (UID: \"115ab490-eada-4fd1-9214-b204629c67d3\") "
Jul 17 01:05:19.081039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:19.081007    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115ab490-eada-4fd1-9214-b204629c67d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "115ab490-eada-4fd1-9214-b204629c67d3" (UID: "115ab490-eada-4fd1-9214-b204629c67d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:05:19.181992 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:19.181958    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115ab490-eada-4fd1-9214-b204629c67d3-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:19.750178 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:19.750140    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4"
Jul 17 01:05:19.750392 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:19.750139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4" event={"ID":"115ab490-eada-4fd1-9214-b204629c67d3","Type":"ContainerDied","Data":"78da401d11bd461145eccf45dd692f30d4f20e48510192ec3146baf41f700e3d"}
Jul 17 01:05:19.750392 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:19.750272    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78da401d11bd461145eccf45dd692f30d4f20e48510192ec3146baf41f700e3d"
Jul 17 01:05:22.962858 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:22.962785    2573 scope.go:117] "RemoveContainer" containerID="b14e358bfb0d6c1f41c3ac03ac6d06f009a92e8361c2618411ab512552a98192"
Jul 17 01:05:23.777260 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:23.777218    2573 generic.go:356] "Generic (PLEG): container finished" podID="b572604b-0c21-42f9-9734-edd55753f99c" containerID="f26d42f6c71e3e32a04feb16d5d2357d657e3b395963ed8d5821b76438b6368d" exitCode=6
Jul 17 01:05:23.777465 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:23.777295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" event={"ID":"b572604b-0c21-42f9-9734-edd55753f99c","Type":"ContainerDied","Data":"f26d42f6c71e3e32a04feb16d5d2357d657e3b395963ed8d5821b76438b6368d"}
Jul 17 01:05:23.777465 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:23.777367    2573 scope.go:117] "RemoveContainer" containerID="b14e358bfb0d6c1f41c3ac03ac6d06f009a92e8361c2618411ab512552a98192"
Jul 17 01:05:23.777754 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:23.777720    2573 scope.go:117] "RemoveContainer" containerID="f26d42f6c71e3e32a04feb16d5d2357d657e3b395963ed8d5821b76438b6368d"
Jul 17 01:05:23.778030 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:23.777994    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-vbxj5_odh-ai-gateway-infra(b572604b-0c21-42f9-9734-edd55753f99c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" podUID="b572604b-0c21-42f9-9734-edd55753f99c"
Jul 17 01:05:24.801659 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:24.801614    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"]
Jul 17 01:05:24.969686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:24.969654    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"
Jul 17 01:05:25.125452 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.125333    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b572604b-0c21-42f9-9734-edd55753f99c-maas-api-service-ca\") pod \"b572604b-0c21-42f9-9734-edd55753f99c\" (UID: \"b572604b-0c21-42f9-9734-edd55753f99c\") "
Jul 17 01:05:25.125810 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.125759    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b572604b-0c21-42f9-9734-edd55753f99c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b572604b-0c21-42f9-9734-edd55753f99c" (UID: "b572604b-0c21-42f9-9734-edd55753f99c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:05:25.226539 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.226493    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b572604b-0c21-42f9-9734-edd55753f99c-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:25.793021 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.792986    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"
Jul 17 01:05:25.793257 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.792985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5" event={"ID":"b572604b-0c21-42f9-9734-edd55753f99c","Type":"ContainerDied","Data":"a7ed2b9ae0a337cb23828f71ad92564ef7b57db3bff0139d2cc4376b4622c71f"}
Jul 17 01:05:25.793257 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.793166    2573 scope.go:117] "RemoveContainer" containerID="f26d42f6c71e3e32a04feb16d5d2357d657e3b395963ed8d5821b76438b6368d"
Jul 17 01:05:25.816207 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.816162    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"]
Jul 17 01:05:25.819627 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.819587    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-vbxj5"]
Jul 17 01:05:25.969720 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:25.969679    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b572604b-0c21-42f9-9734-edd55753f99c" path="/var/lib/kubelet/pods/b572604b-0c21-42f9-9734-edd55753f99c/volumes"
Jul 17 01:05:26.715423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:26.714728    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"]
Jul 17 01:05:26.715423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:26.715024    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch" podUID="77d7f187-cf98-48bb-83a2-19b378328312" containerName="maas-api" containerID="cri-o://1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b" gracePeriod=30
Jul 17 01:05:26.994871 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:26.994833    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:05:27.140728 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.140686    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c6zm\" (UniqueName: \"kubernetes.io/projected/77d7f187-cf98-48bb-83a2-19b378328312-kube-api-access-9c6zm\") pod \"77d7f187-cf98-48bb-83a2-19b378328312\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") "
Jul 17 01:05:27.140994 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.140778    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d7f187-cf98-48bb-83a2-19b378328312-openshift-service-ca\") pod \"77d7f187-cf98-48bb-83a2-19b378328312\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") "
Jul 17 01:05:27.140994 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.140821    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls\") pod \"77d7f187-cf98-48bb-83a2-19b378328312\" (UID: \"77d7f187-cf98-48bb-83a2-19b378328312\") "
Jul 17 01:05:27.141256 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.141223    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d7f187-cf98-48bb-83a2-19b378328312-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77d7f187-cf98-48bb-83a2-19b378328312" (UID: "77d7f187-cf98-48bb-83a2-19b378328312"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:05:27.143185 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.143142    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d7f187-cf98-48bb-83a2-19b378328312-kube-api-access-9c6zm" (OuterVolumeSpecName: "kube-api-access-9c6zm") pod "77d7f187-cf98-48bb-83a2-19b378328312" (UID: "77d7f187-cf98-48bb-83a2-19b378328312"). InnerVolumeSpecName "kube-api-access-9c6zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:05:27.143185 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.143166    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77d7f187-cf98-48bb-83a2-19b378328312" (UID: "77d7f187-cf98-48bb-83a2-19b378328312"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 01:05:27.241623 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.241520    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d7f187-cf98-48bb-83a2-19b378328312-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:27.241623 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.241557    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d7f187-cf98-48bb-83a2-19b378328312-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:27.241623 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.241571    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c6zm\" (UniqueName: \"kubernetes.io/projected/77d7f187-cf98-48bb-83a2-19b378328312-kube-api-access-9c6zm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:05:27.809072 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.809038    2573 generic.go:356] "Generic (PLEG): container finished" podID="77d7f187-cf98-48bb-83a2-19b378328312" containerID="1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b" exitCode=0
Jul 17 01:05:27.809293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.809085    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch" event={"ID":"77d7f187-cf98-48bb-83a2-19b378328312","Type":"ContainerDied","Data":"1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b"}
Jul 17 01:05:27.809293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.809126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch" event={"ID":"77d7f187-cf98-48bb-83a2-19b378328312","Type":"ContainerDied","Data":"2606a9d0ac3b9ed7c4523366a4c183b307b085b16a99d202bb377995d660f18c"}
Jul 17 01:05:27.809293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.809147    2573 scope.go:117] "RemoveContainer" containerID="1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b"
Jul 17 01:05:27.809293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.809088    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"
Jul 17 01:05:27.819691 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.819665    2573 scope.go:117] "RemoveContainer" containerID="1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b"
Jul 17 01:05:27.820068 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:27.820046    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b\": container with ID starting with 1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b not found: ID does not exist" containerID="1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b"
Jul 17 01:05:27.820137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.820078    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b"} err="failed to get container status \"1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b\": rpc error: code = NotFound desc = could not find container \"1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b\": container with ID starting with 1bc6343b6002ce3ff73a188dc2c82654c9c615e8b509b55879b61ab15f1e928b not found: ID does not exist"
Jul 17 01:05:27.830473 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.830425    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"]
Jul 17 01:05:27.838888 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.838845    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49fd04-75974997b4-s8nch"]
Jul 17 01:05:27.969305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:27.969261    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d7f187-cf98-48bb-83a2-19b378328312" path="/var/lib/kubelet/pods/77d7f187-cf98-48bb-83a2-19b378328312/volumes"
Jul 17 01:05:43.429520 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429472    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"]
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429899    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429912    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429920    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429925    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429941    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d7f187-cf98-48bb-83a2-19b378328312" containerName="maas-api"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429947    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d7f187-cf98-48bb-83a2-19b378328312" containerName="maas-api"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429958    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115ab490-eada-4fd1-9214-b204629c67d3" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.429964    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="115ab490-eada-4fd1-9214-b204629c67d3" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.430036    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.430043    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="115ab490-eada-4fd1-9214-b204629c67d3" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.430051    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d7f187-cf98-48bb-83a2-19b378328312" containerName="maas-api"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.430057    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.430063    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:43.430137 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.430069    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b859d2a7-ffc5-4434-a3e7-0a98f54a9e9a" containerName="revoke-keys"
Jul 17 01:05:43.434410 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.434382    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"
Jul 17 01:05:43.437206 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.437166    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:05:43.440590 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.440554    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"]
Jul 17 01:05:43.487926 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.487887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9676d4d7-5bca-46b8-98cd-29f4c606e446-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm\" (UID: \"9676d4d7-5bca-46b8-98cd-29f4c606e446\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"
Jul 17 01:05:43.589279 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.589235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9676d4d7-5bca-46b8-98cd-29f4c606e446-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm\" (UID: \"9676d4d7-5bca-46b8-98cd-29f4c606e446\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"
Jul 17 01:05:43.590085 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.590052    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9676d4d7-5bca-46b8-98cd-29f4c606e446-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm\" (UID: \"9676d4d7-5bca-46b8-98cd-29f4c606e446\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"
Jul 17 01:05:43.749538 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.749415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"
Jul 17 01:05:43.905390 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:43.905362    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"]
Jul 17 01:05:43.907563 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:05:43.907529    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9676d4d7_5bca_46b8_98cd_29f4c606e446.slice/crio-1037b0649f2c9086d6bfb231a217131c89a0b46ef4ab967757ba19b4b39bfdb4 WatchSource:0}: Error finding container 1037b0649f2c9086d6bfb231a217131c89a0b46ef4ab967757ba19b4b39bfdb4: Status 404 returned error can't find the container with id 1037b0649f2c9086d6bfb231a217131c89a0b46ef4ab967757ba19b4b39bfdb4
Jul 17 01:05:44.920316 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:44.920213    2573 generic.go:356] "Generic (PLEG): container finished" podID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerID="d7efcba90d6e32b8de5f7469a526c7ddb56fd46b4ebb1dc3fcbccca93fbaf94a" exitCode=6
Jul 17 01:05:44.920316 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:44.920295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" event={"ID":"9676d4d7-5bca-46b8-98cd-29f4c606e446","Type":"ContainerDied","Data":"d7efcba90d6e32b8de5f7469a526c7ddb56fd46b4ebb1dc3fcbccca93fbaf94a"}
Jul 17 01:05:44.920914 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:44.920358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" event={"ID":"9676d4d7-5bca-46b8-98cd-29f4c606e446","Type":"ContainerStarted","Data":"1037b0649f2c9086d6bfb231a217131c89a0b46ef4ab967757ba19b4b39bfdb4"}
Jul 17 01:05:44.920914 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:44.920564    2573 scope.go:117] "RemoveContainer" containerID="d7efcba90d6e32b8de5f7469a526c7ddb56fd46b4ebb1dc3fcbccca93fbaf94a"
Jul 17 01:05:45.929224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:45.929116    2573 generic.go:356] "Generic (PLEG): container finished" podID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerID="75363e795be3b1246b5d15e9bc36f943e425b012f66de4dcd8408c2bc2d14a7d" exitCode=6
Jul 17 01:05:45.929224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:45.929160    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" event={"ID":"9676d4d7-5bca-46b8-98cd-29f4c606e446","Type":"ContainerDied","Data":"75363e795be3b1246b5d15e9bc36f943e425b012f66de4dcd8408c2bc2d14a7d"}
Jul 17 01:05:45.929775 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:45.929238    2573 scope.go:117] "RemoveContainer" containerID="d7efcba90d6e32b8de5f7469a526c7ddb56fd46b4ebb1dc3fcbccca93fbaf94a"
Jul 17 01:05:45.929775 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:45.929460    2573 scope.go:117] "RemoveContainer" containerID="75363e795be3b1246b5d15e9bc36f943e425b012f66de4dcd8408c2bc2d14a7d"
Jul 17 01:05:45.929775 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:45.929759    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm_odh-ai-gateway-infra(9676d4d7-5bca-46b8-98cd-29f4c606e446)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446"
Jul 17 01:05:46.399880 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.399834    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"]
Jul 17 01:05:46.400295 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.400282    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:46.400342 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.400296    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b572604b-0c21-42f9-9734-edd55753f99c" containerName="revoke-keys"
Jul 17 01:05:46.404809 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.404765    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.406860 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.406821    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e1dd24-serving-cert\""
Jul 17 01:05:46.406860 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.406846    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlwm6\""
Jul 17 01:05:46.412185 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.412149    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"]
Jul 17 01:05:46.522200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.522154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdvx9\" (UniqueName: \"kubernetes.io/projected/09d3b57a-e993-43b7-ae29-da8a4352e731-kube-api-access-pdvx9\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.522490 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.522229    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d3b57a-e993-43b7-ae29-da8a4352e731-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.522490 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.522360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d3b57a-e993-43b7-ae29-da8a4352e731-maas-api-tls\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.623510 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.623461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d3b57a-e993-43b7-ae29-da8a4352e731-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.623686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.623536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d3b57a-e993-43b7-ae29-da8a4352e731-maas-api-tls\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.623686 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.623585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdvx9\" (UniqueName: \"kubernetes.io/projected/09d3b57a-e993-43b7-ae29-da8a4352e731-kube-api-access-pdvx9\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.624175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.624150    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d3b57a-e993-43b7-ae29-da8a4352e731-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.626373 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.626340    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d3b57a-e993-43b7-ae29-da8a4352e731-maas-api-tls\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.631202 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.631166    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdvx9\" (UniqueName: \"kubernetes.io/projected/09d3b57a-e993-43b7-ae29-da8a4352e731-kube-api-access-pdvx9\") pod \"maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.719530 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.719420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:46.877829 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.877773    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"]
Jul 17 01:05:46.879051 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:05:46.879002    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d3b57a_e993_43b7_ae29_da8a4352e731.slice/crio-92774de6675a78a0225f38dfecba9807c624b2ac8ece0d86e82573f7275f0e1f WatchSource:0}: Error finding container 92774de6675a78a0225f38dfecba9807c624b2ac8ece0d86e82573f7275f0e1f: Status 404 returned error can't find the container with id 92774de6675a78a0225f38dfecba9807c624b2ac8ece0d86e82573f7275f0e1f
Jul 17 01:05:46.937372 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.937318    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq" event={"ID":"09d3b57a-e993-43b7-ae29-da8a4352e731","Type":"ContainerStarted","Data":"92774de6675a78a0225f38dfecba9807c624b2ac8ece0d86e82573f7275f0e1f"}
Jul 17 01:05:46.939677 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:46.939647    2573 scope.go:117] "RemoveContainer" containerID="75363e795be3b1246b5d15e9bc36f943e425b012f66de4dcd8408c2bc2d14a7d"
Jul 17 01:05:46.939998 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:46.939972    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm_odh-ai-gateway-infra(9676d4d7-5bca-46b8-98cd-29f4c606e446)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446"
Jul 17 01:05:47.948634 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:47.948592    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq" event={"ID":"09d3b57a-e993-43b7-ae29-da8a4352e731","Type":"ContainerStarted","Data":"cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3"}
Jul 17 01:05:47.949142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:47.948667    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:47.979430 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:47.979370    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq" podStartSLOduration=1.5782553689999999 podStartE2EDuration="1.979346712s" totalImagesPullingTime="401.091343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:05:47.97691385 +0000 UTC m=+2942.553066424" watchObservedRunningTime="2026-07-17 01:05:47.979346712 +0000 UTC m=+2942.555499286"
Jul 17 01:05:49.156880 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.156827    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"]
Jul 17 01:05:49.160828 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.160786    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"
Jul 17 01:05:49.167079 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.167036    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"]
Jul 17 01:05:49.250703 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.250651    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857a6ae6-dff5-4e68-a940-623ec930def8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-7589b\" (UID: \"857a6ae6-dff5-4e68-a940-623ec930def8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"
Jul 17 01:05:49.352242 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.352190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857a6ae6-dff5-4e68-a940-623ec930def8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-7589b\" (UID: \"857a6ae6-dff5-4e68-a940-623ec930def8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"
Jul 17 01:05:49.352975 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.352943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857a6ae6-dff5-4e68-a940-623ec930def8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-7589b\" (UID: \"857a6ae6-dff5-4e68-a940-623ec930def8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"
Jul 17 01:05:49.476556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.476441    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"
Jul 17 01:05:49.636546 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.636487    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"]
Jul 17 01:05:49.638950 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:05:49.638917    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857a6ae6_dff5_4e68_a940_623ec930def8.slice/crio-4384f1559e3528b13592899c8820239bce86554abec7426f0302197651057193 WatchSource:0}: Error finding container 4384f1559e3528b13592899c8820239bce86554abec7426f0302197651057193: Status 404 returned error can't find the container with id 4384f1559e3528b13592899c8820239bce86554abec7426f0302197651057193
Jul 17 01:05:49.969001 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.968966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" event={"ID":"857a6ae6-dff5-4e68-a940-623ec930def8","Type":"ContainerStarted","Data":"c73fc5c5ff6b6353bb55dd09eccf23bdf90575c7016ccba2994c2b1d30214c02"}
Jul 17 01:05:49.969205 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.969025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" event={"ID":"857a6ae6-dff5-4e68-a940-623ec930def8","Type":"ContainerStarted","Data":"4384f1559e3528b13592899c8820239bce86554abec7426f0302197651057193"}
Jul 17 01:05:49.978071 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:49.978007    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" podStartSLOduration=0.977988002 podStartE2EDuration="977.988002ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:05:49.974890836 +0000 UTC m=+2944.551043411" watchObservedRunningTime="2026-07-17 01:05:49.977988002 +0000 UTC m=+2944.554140576"
Jul 17 01:05:50.971576 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:50.971537    2573 generic.go:356] "Generic (PLEG): container finished" podID="857a6ae6-dff5-4e68-a940-623ec930def8" containerID="c73fc5c5ff6b6353bb55dd09eccf23bdf90575c7016ccba2994c2b1d30214c02" exitCode=6
Jul 17 01:05:50.972096 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:50.971626    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" event={"ID":"857a6ae6-dff5-4e68-a940-623ec930def8","Type":"ContainerDied","Data":"c73fc5c5ff6b6353bb55dd09eccf23bdf90575c7016ccba2994c2b1d30214c02"}
Jul 17 01:05:50.972096 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:50.971842    2573 scope.go:117] "RemoveContainer" containerID="c73fc5c5ff6b6353bb55dd09eccf23bdf90575c7016ccba2994c2b1d30214c02"
Jul 17 01:05:51.011420 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.011368    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"]
Jul 17 01:05:51.016490 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.016460    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"
Jul 17 01:05:51.021494 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.021452    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"]
Jul 17 01:05:51.167775 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.167724    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a141cd-686b-472b-8c92-f74f07c7ed64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb\" (UID: \"a5a141cd-686b-472b-8c92-f74f07c7ed64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"
Jul 17 01:05:51.268419 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.268371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a141cd-686b-472b-8c92-f74f07c7ed64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb\" (UID: \"a5a141cd-686b-472b-8c92-f74f07c7ed64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"
Jul 17 01:05:51.269127 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.269106    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a141cd-686b-472b-8c92-f74f07c7ed64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb\" (UID: \"a5a141cd-686b-472b-8c92-f74f07c7ed64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"
Jul 17 01:05:51.331903 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.331853    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"
Jul 17 01:05:51.497749 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.497716    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"]
Jul 17 01:05:51.500057 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:05:51.500019    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a141cd_686b_472b_8c92_f74f07c7ed64.slice/crio-6ec5dae25a3117f82ae0c2da8d85ea426295462db262813f25037b5ff47b3e37 WatchSource:0}: Error finding container 6ec5dae25a3117f82ae0c2da8d85ea426295462db262813f25037b5ff47b3e37: Status 404 returned error can't find the container with id 6ec5dae25a3117f82ae0c2da8d85ea426295462db262813f25037b5ff47b3e37
Jul 17 01:05:51.981316 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.981268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" event={"ID":"a5a141cd-686b-472b-8c92-f74f07c7ed64","Type":"ContainerStarted","Data":"3d26d8ffe3ebcc05b2bebd0b94623094e87aea40fb648f042a1751d2aaf83cac"}
Jul 17 01:05:51.981983 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.981954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" event={"ID":"a5a141cd-686b-472b-8c92-f74f07c7ed64","Type":"ContainerStarted","Data":"6ec5dae25a3117f82ae0c2da8d85ea426295462db262813f25037b5ff47b3e37"}
Jul 17 01:05:51.984939 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.984893    2573 generic.go:356] "Generic (PLEG): container finished" podID="857a6ae6-dff5-4e68-a940-623ec930def8" containerID="3280c3ceef580fa28ba34367f001b312f485d4e85945f15b4f933243668e665c" exitCode=6
Jul 17 01:05:51.985163 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.984947    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" event={"ID":"857a6ae6-dff5-4e68-a940-623ec930def8","Type":"ContainerDied","Data":"3280c3ceef580fa28ba34367f001b312f485d4e85945f15b4f933243668e665c"}
Jul 17 01:05:51.985163 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.985020    2573 scope.go:117] "RemoveContainer" containerID="c73fc5c5ff6b6353bb55dd09eccf23bdf90575c7016ccba2994c2b1d30214c02"
Jul 17 01:05:51.985356 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.985202    2573 scope.go:117] "RemoveContainer" containerID="3280c3ceef580fa28ba34367f001b312f485d4e85945f15b4f933243668e665c"
Jul 17 01:05:51.985466 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:51.985445    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-7589b_odh-ai-gateway-infra(857a6ae6-dff5-4e68-a940-623ec930def8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" podUID="857a6ae6-dff5-4e68-a940-623ec930def8"
Jul 17 01:05:51.996848 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:51.996767    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" podStartSLOduration=1.996747263 podStartE2EDuration="1.996747263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:05:51.995194505 +0000 UTC m=+2946.571347080" watchObservedRunningTime="2026-07-17 01:05:51.996747263 +0000 UTC m=+2946.572899836"
Jul 17 01:05:52.993182 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:52.993151    2573 scope.go:117] "RemoveContainer" containerID="3280c3ceef580fa28ba34367f001b312f485d4e85945f15b4f933243668e665c"
Jul 17 01:05:52.993690 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:52.993432    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-7589b_odh-ai-gateway-infra(857a6ae6-dff5-4e68-a940-623ec930def8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" podUID="857a6ae6-dff5-4e68-a940-623ec930def8"
Jul 17 01:05:52.994598 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:52.994568    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerID="3d26d8ffe3ebcc05b2bebd0b94623094e87aea40fb648f042a1751d2aaf83cac" exitCode=6
Jul 17 01:05:52.994737 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:52.994640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" event={"ID":"a5a141cd-686b-472b-8c92-f74f07c7ed64","Type":"ContainerDied","Data":"3d26d8ffe3ebcc05b2bebd0b94623094e87aea40fb648f042a1751d2aaf83cac"}
Jul 17 01:05:52.994830 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:52.994813    2573 scope.go:117] "RemoveContainer" containerID="3d26d8ffe3ebcc05b2bebd0b94623094e87aea40fb648f042a1751d2aaf83cac"
Jul 17 01:05:53.969841 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:53.969724    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:05:54.005038 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.004993    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerID="5560b711a0f0d2071d2e3c58197924b127e36b5646b03938121e38929df2ac2b" exitCode=6
Jul 17 01:05:54.005534 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.005049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" event={"ID":"a5a141cd-686b-472b-8c92-f74f07c7ed64","Type":"ContainerDied","Data":"5560b711a0f0d2071d2e3c58197924b127e36b5646b03938121e38929df2ac2b"}
Jul 17 01:05:54.005534 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.005119    2573 scope.go:117] "RemoveContainer" containerID="3d26d8ffe3ebcc05b2bebd0b94623094e87aea40fb648f042a1751d2aaf83cac"
Jul 17 01:05:54.005534 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.005367    2573 scope.go:117] "RemoveContainer" containerID="5560b711a0f0d2071d2e3c58197924b127e36b5646b03938121e38929df2ac2b"
Jul 17 01:05:54.005708 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:54.005672    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb_odh-ai-gateway-infra(a5a141cd-686b-472b-8c92-f74f07c7ed64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64"
Jul 17 01:05:54.395163 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.395118    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"]
Jul 17 01:05:54.400091 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.400061    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.402024 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.401992    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ac54e2-serving-cert\""
Jul 17 01:05:54.409345 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.409304    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"]
Jul 17 01:05:54.501371 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.501335    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.501371 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.501389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjz2\" (UniqueName: \"kubernetes.io/projected/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-kube-api-access-zbjz2\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.501720 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.501487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.602303 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.602260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.602303 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.602307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjz2\" (UniqueName: \"kubernetes.io/projected/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-kube-api-access-zbjz2\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.602580 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.602349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.602580 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:54.602422    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-serving-cert: secret "maas-api-e2e-shared-b-ac54e2-serving-cert" not found
Jul 17 01:05:54.602580 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:54.602502    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls podName:9dbabf49-66cb-4b79-8d3d-05d226d7c22e nodeName:}" failed. No retries permitted until 2026-07-17 01:05:55.102479162 +0000 UTC m=+2949.678631715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls") pod "maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" (UID: "9dbabf49-66cb-4b79-8d3d-05d226d7c22e") : secret "maas-api-e2e-shared-b-ac54e2-serving-cert" not found
Jul 17 01:05:54.603174 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.603153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:54.612815 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:54.612757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjz2\" (UniqueName: \"kubernetes.io/projected/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-kube-api-access-zbjz2\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:55.013615 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:55.013580    2573 scope.go:117] "RemoveContainer" containerID="5560b711a0f0d2071d2e3c58197924b127e36b5646b03938121e38929df2ac2b"
Jul 17 01:05:55.014163 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:05:55.013963    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb_odh-ai-gateway-infra(a5a141cd-686b-472b-8c92-f74f07c7ed64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64"
Jul 17 01:05:55.108849 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:55.108784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:55.111414 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:55.111389    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:55.315194 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:55.315087    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:55.479584 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:55.479544    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"]
Jul 17 01:05:55.481286 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:05:55.481253    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbabf49_66cb_4b79_8d3d_05d226d7c22e.slice/crio-e42b529db054261aa7754c4e7f3ff8858ba92e132fce5c5270926a0524698849 WatchSource:0}: Error finding container e42b529db054261aa7754c4e7f3ff8858ba92e132fce5c5270926a0524698849: Status 404 returned error can't find the container with id e42b529db054261aa7754c4e7f3ff8858ba92e132fce5c5270926a0524698849
Jul 17 01:05:56.021572 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:56.021532    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" event={"ID":"9dbabf49-66cb-4b79-8d3d-05d226d7c22e","Type":"ContainerStarted","Data":"e42b529db054261aa7754c4e7f3ff8858ba92e132fce5c5270926a0524698849"}
Jul 17 01:05:57.030456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:57.030410    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" event={"ID":"9dbabf49-66cb-4b79-8d3d-05d226d7c22e","Type":"ContainerStarted","Data":"3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd"}
Jul 17 01:05:57.030996 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:57.030497    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:05:57.048092 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:05:57.048026    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" podStartSLOduration=2.52316512 podStartE2EDuration="3.048004858s" totalImagesPullingTime="524.839738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:05:57.044702834 +0000 UTC m=+2951.620855407" watchObservedRunningTime="2026-07-17 01:05:57.048004858 +0000 UTC m=+2951.624157433"
Jul 17 01:06:00.963030 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:00.962993    2573 scope.go:117] "RemoveContainer" containerID="75363e795be3b1246b5d15e9bc36f943e425b012f66de4dcd8408c2bc2d14a7d"
Jul 17 01:06:02.065246 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:02.065206    2573 generic.go:356] "Generic (PLEG): container finished" podID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerID="ed02870e86a14178e38cc3dfa6ea3ba00c158d1625cb239c62be2c7ec49dd588" exitCode=6
Jul 17 01:06:02.065697 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:02.065281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" event={"ID":"9676d4d7-5bca-46b8-98cd-29f4c606e446","Type":"ContainerDied","Data":"ed02870e86a14178e38cc3dfa6ea3ba00c158d1625cb239c62be2c7ec49dd588"}
Jul 17 01:06:02.065697 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:02.065389    2573 scope.go:117] "RemoveContainer" containerID="75363e795be3b1246b5d15e9bc36f943e425b012f66de4dcd8408c2bc2d14a7d"
Jul 17 01:06:02.065875 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:02.065750    2573 scope.go:117] "RemoveContainer" containerID="ed02870e86a14178e38cc3dfa6ea3ba00c158d1625cb239c62be2c7ec49dd588"
Jul 17 01:06:02.066050 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:02.066025    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm_odh-ai-gateway-infra(9676d4d7-5bca-46b8-98cd-29f4c606e446)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446"
Jul 17 01:06:03.043129 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:03.043093    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:06:03.088214 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:03.088179    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"]
Jul 17 01:06:03.243759 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:03.243721    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"
Jul 17 01:06:03.299584 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:03.299486    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9676d4d7-5bca-46b8-98cd-29f4c606e446-maas-api-service-ca\") pod \"9676d4d7-5bca-46b8-98cd-29f4c606e446\" (UID: \"9676d4d7-5bca-46b8-98cd-29f4c606e446\") "
Jul 17 01:06:03.299892 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:03.299868    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9676d4d7-5bca-46b8-98cd-29f4c606e446-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9676d4d7-5bca-46b8-98cd-29f4c606e446" (UID: "9676d4d7-5bca-46b8-98cd-29f4c606e446"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:06:03.400240 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:03.400200    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9676d4d7-5bca-46b8-98cd-29f4c606e446-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:06:04.083857 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:04.083823    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"
Jul 17 01:06:04.083857 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:04.083822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm" event={"ID":"9676d4d7-5bca-46b8-98cd-29f4c606e446","Type":"ContainerDied","Data":"1037b0649f2c9086d6bfb231a217131c89a0b46ef4ab967757ba19b4b39bfdb4"}
Jul 17 01:06:04.084212 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:04.083905    2573 scope.go:117] "RemoveContainer" containerID="ed02870e86a14178e38cc3dfa6ea3ba00c158d1625cb239c62be2c7ec49dd588"
Jul 17 01:06:04.103436 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:04.103393    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"]
Jul 17 01:06:04.107179 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:04.107142    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-hjzrm"]
Jul 17 01:06:05.970927 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:05.970890    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" path="/var/lib/kubelet/pods/9676d4d7-5bca-46b8-98cd-29f4c606e446/volumes"
Jul 17 01:06:06.962512 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:06.962477    2573 scope.go:117] "RemoveContainer" containerID="5560b711a0f0d2071d2e3c58197924b127e36b5646b03938121e38929df2ac2b"
Jul 17 01:06:06.962716 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:06.962670    2573 scope.go:117] "RemoveContainer" containerID="3280c3ceef580fa28ba34367f001b312f485d4e85945f15b4f933243668e665c"
Jul 17 01:06:08.114591 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.114554    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerID="0e65ac3699181ba0db89f58d79bd6b5c3f04ecd0e1a59e46753b5e5a78358cc2" exitCode=6
Jul 17 01:06:08.115116 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.114613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" event={"ID":"a5a141cd-686b-472b-8c92-f74f07c7ed64","Type":"ContainerDied","Data":"0e65ac3699181ba0db89f58d79bd6b5c3f04ecd0e1a59e46753b5e5a78358cc2"}
Jul 17 01:06:08.115116 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.114704    2573 scope.go:117] "RemoveContainer" containerID="5560b711a0f0d2071d2e3c58197924b127e36b5646b03938121e38929df2ac2b"
Jul 17 01:06:08.115116 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.115010    2573 scope.go:117] "RemoveContainer" containerID="0e65ac3699181ba0db89f58d79bd6b5c3f04ecd0e1a59e46753b5e5a78358cc2"
Jul 17 01:06:08.115307 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:08.115279    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb_odh-ai-gateway-infra(a5a141cd-686b-472b-8c92-f74f07c7ed64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64"
Jul 17 01:06:08.116920 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.116896    2573 generic.go:356] "Generic (PLEG): container finished" podID="857a6ae6-dff5-4e68-a940-623ec930def8" containerID="18633b09de8d7fd4836bb2fe7dba823003156090cd58f130a0cb65b7552e07c6" exitCode=6
Jul 17 01:06:08.117020 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.116955    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" event={"ID":"857a6ae6-dff5-4e68-a940-623ec930def8","Type":"ContainerDied","Data":"18633b09de8d7fd4836bb2fe7dba823003156090cd58f130a0cb65b7552e07c6"}
Jul 17 01:06:08.117313 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.117297    2573 scope.go:117] "RemoveContainer" containerID="18633b09de8d7fd4836bb2fe7dba823003156090cd58f130a0cb65b7552e07c6"
Jul 17 01:06:08.117513 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:08.117496    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-7589b_odh-ai-gateway-infra(857a6ae6-dff5-4e68-a940-623ec930def8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" podUID="857a6ae6-dff5-4e68-a940-623ec930def8"
Jul 17 01:06:08.127157 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:08.127134    2573 scope.go:117] "RemoveContainer" containerID="3280c3ceef580fa28ba34367f001b312f485d4e85945f15b4f933243668e665c"
Jul 17 01:06:09.140362 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.140325    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"]
Jul 17 01:06:09.151196 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.151154    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"]
Jul 17 01:06:09.297020 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.296993    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"
Jul 17 01:06:09.328958 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.328923    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"
Jul 17 01:06:09.360707 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.360669    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857a6ae6-dff5-4e68-a940-623ec930def8-maas-api-service-ca\") pod \"857a6ae6-dff5-4e68-a940-623ec930def8\" (UID: \"857a6ae6-dff5-4e68-a940-623ec930def8\") "
Jul 17 01:06:09.360900 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.360716    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a141cd-686b-472b-8c92-f74f07c7ed64-maas-api-service-ca\") pod \"a5a141cd-686b-472b-8c92-f74f07c7ed64\" (UID: \"a5a141cd-686b-472b-8c92-f74f07c7ed64\") "
Jul 17 01:06:09.361143 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.361122    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a141cd-686b-472b-8c92-f74f07c7ed64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5a141cd-686b-472b-8c92-f74f07c7ed64" (UID: "a5a141cd-686b-472b-8c92-f74f07c7ed64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:06:09.361193 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.361142    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857a6ae6-dff5-4e68-a940-623ec930def8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "857a6ae6-dff5-4e68-a940-623ec930def8" (UID: "857a6ae6-dff5-4e68-a940-623ec930def8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:06:09.462309 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.462214    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857a6ae6-dff5-4e68-a940-623ec930def8-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:06:09.462309 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:09.462247    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a141cd-686b-472b-8c92-f74f07c7ed64-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:06:10.136376 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.136343    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"
Jul 17 01:06:10.136597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.136377    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b" event={"ID":"857a6ae6-dff5-4e68-a940-623ec930def8","Type":"ContainerDied","Data":"4384f1559e3528b13592899c8820239bce86554abec7426f0302197651057193"}
Jul 17 01:06:10.136597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.136447    2573 scope.go:117] "RemoveContainer" containerID="18633b09de8d7fd4836bb2fe7dba823003156090cd58f130a0cb65b7552e07c6"
Jul 17 01:06:10.137819 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.137781    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb" event={"ID":"a5a141cd-686b-472b-8c92-f74f07c7ed64","Type":"ContainerDied","Data":"6ec5dae25a3117f82ae0c2da8d85ea426295462db262813f25037b5ff47b3e37"}
Jul 17 01:06:10.137934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.137829    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"
Jul 17 01:06:10.145985 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.145757    2573 scope.go:117] "RemoveContainer" containerID="0e65ac3699181ba0db89f58d79bd6b5c3f04ecd0e1a59e46753b5e5a78358cc2"
Jul 17 01:06:10.157520 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.157488    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"]
Jul 17 01:06:10.162699 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.162674    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-ms8vb"]
Jul 17 01:06:10.172852 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.172822    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"]
Jul 17 01:06:10.175356 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:10.175330    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7589b"]
Jul 17 01:06:11.968959 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:11.968922    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" path="/var/lib/kubelet/pods/857a6ae6-dff5-4e68-a940-623ec930def8/volumes"
Jul 17 01:06:11.969328 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:11.969278    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" path="/var/lib/kubelet/pods/a5a141cd-686b-472b-8c92-f74f07c7ed64/volumes"
Jul 17 01:06:43.479319 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479281    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"]
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479658    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479670    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479690    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479696    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479702    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479708    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479716    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479724    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479734    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479740    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479748    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479754    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479763    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479768    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479776    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.479844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479782    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.480556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479870    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:43.480556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479879    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.480556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479887    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.480556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479898    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:43.480556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479904    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.480556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479910    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="857a6ae6-dff5-4e68-a940-623ec930def8" containerName="revoke-keys"
Jul 17 01:06:43.480556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.479917    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:43.482954 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.482937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"
Jul 17 01:06:43.484951 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.484929    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:06:43.489139 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.489118    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"]
Jul 17 01:06:43.625041 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.625008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972b718d-6cb4-4e0c-8f95-dcf140870d57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht\" (UID: \"972b718d-6cb4-4e0c-8f95-dcf140870d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"
Jul 17 01:06:43.726454 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.726421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972b718d-6cb4-4e0c-8f95-dcf140870d57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht\" (UID: \"972b718d-6cb4-4e0c-8f95-dcf140870d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"
Jul 17 01:06:43.727032 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.727012    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972b718d-6cb4-4e0c-8f95-dcf140870d57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht\" (UID: \"972b718d-6cb4-4e0c-8f95-dcf140870d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"
Jul 17 01:06:43.795149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.795122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"
Jul 17 01:06:43.931856 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:43.931823    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"]
Jul 17 01:06:43.932965 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:06:43.932937    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972b718d_6cb4_4e0c_8f95_dcf140870d57.slice/crio-59ec58c8f07048106789689f6031d188b5123e97635651b83b2d06b6c0028118 WatchSource:0}: Error finding container 59ec58c8f07048106789689f6031d188b5123e97635651b83b2d06b6c0028118: Status 404 returned error can't find the container with id 59ec58c8f07048106789689f6031d188b5123e97635651b83b2d06b6c0028118
Jul 17 01:06:44.333368 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:44.333336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" event={"ID":"972b718d-6cb4-4e0c-8f95-dcf140870d57","Type":"ContainerStarted","Data":"08d2e6ccbb6f40e635171e46edeafdb59752ed8462e014a5080d656950606f53"}
Jul 17 01:06:44.333490 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:44.333392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" event={"ID":"972b718d-6cb4-4e0c-8f95-dcf140870d57","Type":"ContainerStarted","Data":"59ec58c8f07048106789689f6031d188b5123e97635651b83b2d06b6c0028118"}
Jul 17 01:06:44.347554 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:44.347513    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" podStartSLOduration=1.347499356 podStartE2EDuration="1.347499356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:06:44.345579935 +0000 UTC m=+2998.921732509" watchObservedRunningTime="2026-07-17 01:06:44.347499356 +0000 UTC m=+2998.923651930"
Jul 17 01:06:45.340399 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:45.340365    2573 generic.go:356] "Generic (PLEG): container finished" podID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerID="08d2e6ccbb6f40e635171e46edeafdb59752ed8462e014a5080d656950606f53" exitCode=6
Jul 17 01:06:45.340846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:45.340455    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" event={"ID":"972b718d-6cb4-4e0c-8f95-dcf140870d57","Type":"ContainerDied","Data":"08d2e6ccbb6f40e635171e46edeafdb59752ed8462e014a5080d656950606f53"}
Jul 17 01:06:45.340846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:45.340606    2573 scope.go:117] "RemoveContainer" containerID="08d2e6ccbb6f40e635171e46edeafdb59752ed8462e014a5080d656950606f53"
Jul 17 01:06:46.349551 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:46.349517    2573 generic.go:356] "Generic (PLEG): container finished" podID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerID="4a5b86bc06227c0c25c55c511001340a892ece1e3ff0082886ecd7171fab934e" exitCode=6
Jul 17 01:06:46.350049 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:46.349598    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" event={"ID":"972b718d-6cb4-4e0c-8f95-dcf140870d57","Type":"ContainerDied","Data":"4a5b86bc06227c0c25c55c511001340a892ece1e3ff0082886ecd7171fab934e"}
Jul 17 01:06:46.350049 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:46.349655    2573 scope.go:117] "RemoveContainer" containerID="08d2e6ccbb6f40e635171e46edeafdb59752ed8462e014a5080d656950606f53"
Jul 17 01:06:46.350049 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:46.349848    2573 scope.go:117] "RemoveContainer" containerID="4a5b86bc06227c0c25c55c511001340a892ece1e3ff0082886ecd7171fab934e"
Jul 17 01:06:46.350049 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:46.350027    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht_odh-ai-gateway-infra(972b718d-6cb4-4e0c-8f95-dcf140870d57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57"
Jul 17 01:06:47.357288 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:47.357265    2573 scope.go:117] "RemoveContainer" containerID="4a5b86bc06227c0c25c55c511001340a892ece1e3ff0082886ecd7171fab934e"
Jul 17 01:06:47.357726 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:47.357423    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht_odh-ai-gateway-infra(972b718d-6cb4-4e0c-8f95-dcf140870d57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57"
Jul 17 01:06:47.941726 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:47.941624    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 01:06:47.953972 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:47.953948    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 01:06:49.197421 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.197383    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"]
Jul 17 01:06:49.197972 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.197952    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:49.198075 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.197974    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:49.198138 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.198084    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9676d4d7-5bca-46b8-98cd-29f4c606e446" containerName="revoke-keys"
Jul 17 01:06:49.198138 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.198097    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a141cd-686b-472b-8c92-f74f07c7ed64" containerName="revoke-keys"
Jul 17 01:06:49.201043 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.201022    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"
Jul 17 01:06:49.207720 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.207692    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"]
Jul 17 01:06:49.269274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.269249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d20dbcf-9612-4631-84e6-194ca284d04a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q\" (UID: \"3d20dbcf-9612-4631-84e6-194ca284d04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"
Jul 17 01:06:49.370495 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.370467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d20dbcf-9612-4631-84e6-194ca284d04a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q\" (UID: \"3d20dbcf-9612-4631-84e6-194ca284d04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"
Jul 17 01:06:49.371097 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.371080    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d20dbcf-9612-4631-84e6-194ca284d04a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q\" (UID: \"3d20dbcf-9612-4631-84e6-194ca284d04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"
Jul 17 01:06:49.514274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.514253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"
Jul 17 01:06:49.640185 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:49.640157    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"]
Jul 17 01:06:49.642089 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:06:49.642058    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d20dbcf_9612_4631_84e6_194ca284d04a.slice/crio-6204487fb6150d8bb0d3609fc5b616b244b5c61c76fe67a7fb661c5440643551 WatchSource:0}: Error finding container 6204487fb6150d8bb0d3609fc5b616b244b5c61c76fe67a7fb661c5440643551: Status 404 returned error can't find the container with id 6204487fb6150d8bb0d3609fc5b616b244b5c61c76fe67a7fb661c5440643551
Jul 17 01:06:50.378962 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:50.378934    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerID="88111bbe2fa1b102669c855360f4127ba1d57323897ab2d2916ef686464057a8" exitCode=6
Jul 17 01:06:50.379330 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:50.378979    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" event={"ID":"3d20dbcf-9612-4631-84e6-194ca284d04a","Type":"ContainerDied","Data":"88111bbe2fa1b102669c855360f4127ba1d57323897ab2d2916ef686464057a8"}
Jul 17 01:06:50.379330 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:50.379013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" event={"ID":"3d20dbcf-9612-4631-84e6-194ca284d04a","Type":"ContainerStarted","Data":"6204487fb6150d8bb0d3609fc5b616b244b5c61c76fe67a7fb661c5440643551"}
Jul 17 01:06:50.379330 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:50.379165    2573 scope.go:117] "RemoveContainer" containerID="88111bbe2fa1b102669c855360f4127ba1d57323897ab2d2916ef686464057a8"
Jul 17 01:06:51.386730 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:51.386697    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerID="940a9e6b4cd522d03d0ee39767f8526e51815d0fcb050652b8205335a0ddd142" exitCode=6
Jul 17 01:06:51.387197 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:51.386785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" event={"ID":"3d20dbcf-9612-4631-84e6-194ca284d04a","Type":"ContainerDied","Data":"940a9e6b4cd522d03d0ee39767f8526e51815d0fcb050652b8205335a0ddd142"}
Jul 17 01:06:51.387197 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:51.386868    2573 scope.go:117] "RemoveContainer" containerID="88111bbe2fa1b102669c855360f4127ba1d57323897ab2d2916ef686464057a8"
Jul 17 01:06:51.387197 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:51.387003    2573 scope.go:117] "RemoveContainer" containerID="940a9e6b4cd522d03d0ee39767f8526e51815d0fcb050652b8205335a0ddd142"
Jul 17 01:06:51.387320 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:51.387207    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q_odh-ai-gateway-infra(3d20dbcf-9612-4631-84e6-194ca284d04a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a"
Jul 17 01:06:52.394987 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:52.394961    2573 scope.go:117] "RemoveContainer" containerID="940a9e6b4cd522d03d0ee39767f8526e51815d0fcb050652b8205335a0ddd142"
Jul 17 01:06:52.395351 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:52.395119    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q_odh-ai-gateway-infra(3d20dbcf-9612-4631-84e6-194ca284d04a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a"
Jul 17 01:06:56.513879 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.513836    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"]
Jul 17 01:06:56.517066 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.517049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"
Jul 17 01:06:56.523849 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.523626    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"]
Jul 17 01:06:56.626006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.625738    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcb467-6931-44c6-b0fb-e359f690ebf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5nmb2\" (UID: \"46dcb467-6931-44c6-b0fb-e359f690ebf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"
Jul 17 01:06:56.726301 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.726276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcb467-6931-44c6-b0fb-e359f690ebf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5nmb2\" (UID: \"46dcb467-6931-44c6-b0fb-e359f690ebf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"
Jul 17 01:06:56.726889 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.726864    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcb467-6931-44c6-b0fb-e359f690ebf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-5nmb2\" (UID: \"46dcb467-6931-44c6-b0fb-e359f690ebf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"
Jul 17 01:06:56.829715 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.829642    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"
Jul 17 01:06:56.975007 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:56.974980    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"]
Jul 17 01:06:56.976306 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:06:56.976277    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dcb467_6931_44c6_b0fb_e359f690ebf3.slice/crio-5d76d455fe44e8be40c3154ef879174486949ab60225b2bba7a53f016077b0d4 WatchSource:0}: Error finding container 5d76d455fe44e8be40c3154ef879174486949ab60225b2bba7a53f016077b0d4: Status 404 returned error can't find the container with id 5d76d455fe44e8be40c3154ef879174486949ab60225b2bba7a53f016077b0d4
Jul 17 01:06:57.424217 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:57.424144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" event={"ID":"46dcb467-6931-44c6-b0fb-e359f690ebf3","Type":"ContainerStarted","Data":"028d3fe26b06122d7573e448a6f60d3d1aaa024baa9eda2669fce6154bf914df"}
Jul 17 01:06:57.424217 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:57.424204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" event={"ID":"46dcb467-6931-44c6-b0fb-e359f690ebf3","Type":"ContainerStarted","Data":"5d76d455fe44e8be40c3154ef879174486949ab60225b2bba7a53f016077b0d4"}
Jul 17 01:06:57.439248 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:57.439204    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" podStartSLOduration=1.439189323 podStartE2EDuration="1.439189323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:06:57.437327071 +0000 UTC m=+3012.013479646" watchObservedRunningTime="2026-07-17 01:06:57.439189323 +0000 UTC m=+3012.015341896"
Jul 17 01:06:58.431276 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:58.431242    2573 generic.go:356] "Generic (PLEG): container finished" podID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerID="028d3fe26b06122d7573e448a6f60d3d1aaa024baa9eda2669fce6154bf914df" exitCode=6
Jul 17 01:06:58.431652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:58.431287    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" event={"ID":"46dcb467-6931-44c6-b0fb-e359f690ebf3","Type":"ContainerDied","Data":"028d3fe26b06122d7573e448a6f60d3d1aaa024baa9eda2669fce6154bf914df"}
Jul 17 01:06:58.431652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:58.431463    2573 scope.go:117] "RemoveContainer" containerID="028d3fe26b06122d7573e448a6f60d3d1aaa024baa9eda2669fce6154bf914df"
Jul 17 01:06:59.438571 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:59.438491    2573 generic.go:356] "Generic (PLEG): container finished" podID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerID="9cf864d716196478ae9f1f7d18a60eaf116de724c8461528387f41f91a53ecb5" exitCode=6
Jul 17 01:06:59.439027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:59.438585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" event={"ID":"46dcb467-6931-44c6-b0fb-e359f690ebf3","Type":"ContainerDied","Data":"9cf864d716196478ae9f1f7d18a60eaf116de724c8461528387f41f91a53ecb5"}
Jul 17 01:06:59.439027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:59.438654    2573 scope.go:117] "RemoveContainer" containerID="028d3fe26b06122d7573e448a6f60d3d1aaa024baa9eda2669fce6154bf914df"
Jul 17 01:06:59.439027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:06:59.438739    2573 scope.go:117] "RemoveContainer" containerID="9cf864d716196478ae9f1f7d18a60eaf116de724c8461528387f41f91a53ecb5"
Jul 17 01:06:59.439027 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:06:59.438953    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5nmb2_odh-ai-gateway-infra(46dcb467-6931-44c6-b0fb-e359f690ebf3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3"
Jul 17 01:07:00.446641 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:00.446614    2573 scope.go:117] "RemoveContainer" containerID="9cf864d716196478ae9f1f7d18a60eaf116de724c8461528387f41f91a53ecb5"
Jul 17 01:07:00.447040 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:00.446859    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5nmb2_odh-ai-gateway-infra(46dcb467-6931-44c6-b0fb-e359f690ebf3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3"
Jul 17 01:07:00.962729 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:00.962700    2573 scope.go:117] "RemoveContainer" containerID="4a5b86bc06227c0c25c55c511001340a892ece1e3ff0082886ecd7171fab934e"
Jul 17 01:07:01.454688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:01.454656    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" event={"ID":"972b718d-6cb4-4e0c-8f95-dcf140870d57","Type":"ContainerStarted","Data":"7a5695f12530fe8075a0ce14d62498990f84ae91b0fe1f5826840b89571038f4"}
Jul 17 01:07:02.462076 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:02.462041    2573 generic.go:356] "Generic (PLEG): container finished" podID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerID="7a5695f12530fe8075a0ce14d62498990f84ae91b0fe1f5826840b89571038f4" exitCode=6
Jul 17 01:07:02.462489 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:02.462113    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" event={"ID":"972b718d-6cb4-4e0c-8f95-dcf140870d57","Type":"ContainerDied","Data":"7a5695f12530fe8075a0ce14d62498990f84ae91b0fe1f5826840b89571038f4"}
Jul 17 01:07:02.462489 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:02.462176    2573 scope.go:117] "RemoveContainer" containerID="4a5b86bc06227c0c25c55c511001340a892ece1e3ff0082886ecd7171fab934e"
Jul 17 01:07:02.462489 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:02.462434    2573 scope.go:117] "RemoveContainer" containerID="7a5695f12530fe8075a0ce14d62498990f84ae91b0fe1f5826840b89571038f4"
Jul 17 01:07:02.462670 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:02.462645    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht_odh-ai-gateway-infra(972b718d-6cb4-4e0c-8f95-dcf140870d57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57"
Jul 17 01:07:02.482665 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:02.482642    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"]
Jul 17 01:07:03.599559 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:03.599532    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"
Jul 17 01:07:03.690890 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:03.690856    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972b718d-6cb4-4e0c-8f95-dcf140870d57-maas-api-service-ca\") pod \"972b718d-6cb4-4e0c-8f95-dcf140870d57\" (UID: \"972b718d-6cb4-4e0c-8f95-dcf140870d57\") "
Jul 17 01:07:03.691185 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:03.691164    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972b718d-6cb4-4e0c-8f95-dcf140870d57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "972b718d-6cb4-4e0c-8f95-dcf140870d57" (UID: "972b718d-6cb4-4e0c-8f95-dcf140870d57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:07:03.792428 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:03.792400    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972b718d-6cb4-4e0c-8f95-dcf140870d57-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:07:03.962334 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:03.962299    2573 scope.go:117] "RemoveContainer" containerID="940a9e6b4cd522d03d0ee39767f8526e51815d0fcb050652b8205335a0ddd142"
Jul 17 01:07:04.476725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:04.476647    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"
Jul 17 01:07:04.476725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:04.476654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht" event={"ID":"972b718d-6cb4-4e0c-8f95-dcf140870d57","Type":"ContainerDied","Data":"59ec58c8f07048106789689f6031d188b5123e97635651b83b2d06b6c0028118"}
Jul 17 01:07:04.476725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:04.476710    2573 scope.go:117] "RemoveContainer" containerID="7a5695f12530fe8075a0ce14d62498990f84ae91b0fe1f5826840b89571038f4"
Jul 17 01:07:04.478741 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:04.478714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" event={"ID":"3d20dbcf-9612-4631-84e6-194ca284d04a","Type":"ContainerStarted","Data":"48d6838423a5bf3ef1d50406cb0f200b3b9f01cf4ca5eca07e7542002e4de8af"}
Jul 17 01:07:04.491102 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:04.491079    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"]
Jul 17 01:07:04.496377 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:04.496356    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-9lqht"]
Jul 17 01:07:04.509117 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:04.509077    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" podStartSLOduration=15.509065016 podStartE2EDuration="15.509065016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:07:04.506701724 +0000 UTC m=+3019.082854297" watchObservedRunningTime="2026-07-17 01:07:04.509065016 +0000 UTC m=+3019.085217608"
Jul 17 01:07:05.486712 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:05.486676    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerID="48d6838423a5bf3ef1d50406cb0f200b3b9f01cf4ca5eca07e7542002e4de8af" exitCode=6
Jul 17 01:07:05.487188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:05.486736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" event={"ID":"3d20dbcf-9612-4631-84e6-194ca284d04a","Type":"ContainerDied","Data":"48d6838423a5bf3ef1d50406cb0f200b3b9f01cf4ca5eca07e7542002e4de8af"}
Jul 17 01:07:05.487188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:05.486810    2573 scope.go:117] "RemoveContainer" containerID="940a9e6b4cd522d03d0ee39767f8526e51815d0fcb050652b8205335a0ddd142"
Jul 17 01:07:05.487188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:05.487080    2573 scope.go:117] "RemoveContainer" containerID="48d6838423a5bf3ef1d50406cb0f200b3b9f01cf4ca5eca07e7542002e4de8af"
Jul 17 01:07:05.487363 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:05.487308    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q_odh-ai-gateway-infra(3d20dbcf-9612-4631-84e6-194ca284d04a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a"
Jul 17 01:07:05.516430 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:05.516411    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"]
Jul 17 01:07:05.972559 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:05.972523    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" path="/var/lib/kubelet/pods/972b718d-6cb4-4e0c-8f95-dcf140870d57/volumes"
Jul 17 01:07:06.635357 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:06.635332    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"
Jul 17 01:07:06.817767 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:06.817743    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d20dbcf-9612-4631-84e6-194ca284d04a-maas-api-service-ca\") pod \"3d20dbcf-9612-4631-84e6-194ca284d04a\" (UID: \"3d20dbcf-9612-4631-84e6-194ca284d04a\") "
Jul 17 01:07:06.818100 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:06.818077    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d20dbcf-9612-4631-84e6-194ca284d04a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d20dbcf-9612-4631-84e6-194ca284d04a" (UID: "3d20dbcf-9612-4631-84e6-194ca284d04a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:07:06.918990 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:06.918969    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d20dbcf-9612-4631-84e6-194ca284d04a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:07:07.500509 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:07.500483    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"
Jul 17 01:07:07.500509 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:07.500491    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q" event={"ID":"3d20dbcf-9612-4631-84e6-194ca284d04a","Type":"ContainerDied","Data":"6204487fb6150d8bb0d3609fc5b616b244b5c61c76fe67a7fb661c5440643551"}
Jul 17 01:07:07.500718 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:07.500552    2573 scope.go:117] "RemoveContainer" containerID="48d6838423a5bf3ef1d50406cb0f200b3b9f01cf4ca5eca07e7542002e4de8af"
Jul 17 01:07:07.522541 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:07.522520    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"]
Jul 17 01:07:07.526039 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:07.526020    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-bvr4q"]
Jul 17 01:07:07.968465 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:07.968433    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" path="/var/lib/kubelet/pods/3d20dbcf-9612-4631-84e6-194ca284d04a/volumes"
Jul 17 01:07:14.963038 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:14.963005    2573 scope.go:117] "RemoveContainer" containerID="9cf864d716196478ae9f1f7d18a60eaf116de724c8461528387f41f91a53ecb5"
Jul 17 01:07:15.548472 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:15.548444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" event={"ID":"46dcb467-6931-44c6-b0fb-e359f690ebf3","Type":"ContainerStarted","Data":"d9da63659d1255bc64f8c1f79a8041bfffd9f23d151b6a8b5a1dae192569ca2d"}
Jul 17 01:07:16.556172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:16.556143    2573 generic.go:356] "Generic (PLEG): container finished" podID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerID="d9da63659d1255bc64f8c1f79a8041bfffd9f23d151b6a8b5a1dae192569ca2d" exitCode=6
Jul 17 01:07:16.556580 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:16.556220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" event={"ID":"46dcb467-6931-44c6-b0fb-e359f690ebf3","Type":"ContainerDied","Data":"d9da63659d1255bc64f8c1f79a8041bfffd9f23d151b6a8b5a1dae192569ca2d"}
Jul 17 01:07:16.556580 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:16.556280    2573 scope.go:117] "RemoveContainer" containerID="9cf864d716196478ae9f1f7d18a60eaf116de724c8461528387f41f91a53ecb5"
Jul 17 01:07:16.556580 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:16.556494    2573 scope.go:117] "RemoveContainer" containerID="d9da63659d1255bc64f8c1f79a8041bfffd9f23d151b6a8b5a1dae192569ca2d"
Jul 17 01:07:16.556728 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:16.556682    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-5nmb2_odh-ai-gateway-infra(46dcb467-6931-44c6-b0fb-e359f690ebf3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3"
Jul 17 01:07:16.571870 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:16.571850    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"]
Jul 17 01:07:17.713279 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:17.713255    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"
Jul 17 01:07:17.803022 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:17.802995    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcb467-6931-44c6-b0fb-e359f690ebf3-maas-api-service-ca\") pod \"46dcb467-6931-44c6-b0fb-e359f690ebf3\" (UID: \"46dcb467-6931-44c6-b0fb-e359f690ebf3\") "
Jul 17 01:07:17.803335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:17.803314    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dcb467-6931-44c6-b0fb-e359f690ebf3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46dcb467-6931-44c6-b0fb-e359f690ebf3" (UID: "46dcb467-6931-44c6-b0fb-e359f690ebf3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:07:17.903868 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:17.903782    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcb467-6931-44c6-b0fb-e359f690ebf3-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:07:18.576211 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:18.576179    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"
Jul 17 01:07:18.576487 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:18.576185    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2" event={"ID":"46dcb467-6931-44c6-b0fb-e359f690ebf3","Type":"ContainerDied","Data":"5d76d455fe44e8be40c3154ef879174486949ab60225b2bba7a53f016077b0d4"}
Jul 17 01:07:18.576487 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:18.576322    2573 scope.go:117] "RemoveContainer" containerID="d9da63659d1255bc64f8c1f79a8041bfffd9f23d151b6a8b5a1dae192569ca2d"
Jul 17 01:07:18.595509 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:18.595486    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"]
Jul 17 01:07:18.598025 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:18.598002    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-5nmb2"]
Jul 17 01:07:19.968235 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:19.968192    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" path="/var/lib/kubelet/pods/46dcb467-6931-44c6-b0fb-e359f690ebf3/volumes"
Jul 17 01:07:31.090264 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090226    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"]
Jul 17 01:07:31.090637 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090605    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:31.090637 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090617    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:31.090637 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090631    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090640    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090649    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090655    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090664    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090670    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090681    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090686    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090699    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090704    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090713    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:31.090746 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090722    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:31.091103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090781    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:31.091103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090805    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.091103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090815    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:31.091103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090821    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:31.091103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090827    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:31.091103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090834    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:31.091103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.090840    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:31.093971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.093954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"
Jul 17 01:07:31.096067 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.096046    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:07:31.103927 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.103903    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"]
Jul 17 01:07:31.206338 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.206313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a57e2e-3399-424f-b463-62d4fc851e27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj\" (UID: \"27a57e2e-3399-424f-b463-62d4fc851e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"
Jul 17 01:07:31.307236 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.307199    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a57e2e-3399-424f-b463-62d4fc851e27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj\" (UID: \"27a57e2e-3399-424f-b463-62d4fc851e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"
Jul 17 01:07:31.307864 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.307840    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a57e2e-3399-424f-b463-62d4fc851e27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj\" (UID: \"27a57e2e-3399-424f-b463-62d4fc851e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"
Jul 17 01:07:31.405218 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.405159    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"
Jul 17 01:07:31.535776 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:07:31.535750    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a57e2e_3399_424f_b463_62d4fc851e27.slice/crio-b4c454d269337d9107a765377449823275922e8fdfdcd79aca6111cb3ede8ca8 WatchSource:0}: Error finding container b4c454d269337d9107a765377449823275922e8fdfdcd79aca6111cb3ede8ca8: Status 404 returned error can't find the container with id b4c454d269337d9107a765377449823275922e8fdfdcd79aca6111cb3ede8ca8
Jul 17 01:07:31.536517 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.536477    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"]
Jul 17 01:07:31.651517 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:31.651490    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" event={"ID":"27a57e2e-3399-424f-b463-62d4fc851e27","Type":"ContainerStarted","Data":"b4c454d269337d9107a765377449823275922e8fdfdcd79aca6111cb3ede8ca8"}
Jul 17 01:07:32.659983 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:32.659945    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a57e2e-3399-424f-b463-62d4fc851e27" containerID="9accccc6e1c84e998f0722ab281bcff9270603e49621de48a106cff8dceb0135" exitCode=6
Jul 17 01:07:32.660335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:32.660025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" event={"ID":"27a57e2e-3399-424f-b463-62d4fc851e27","Type":"ContainerDied","Data":"9accccc6e1c84e998f0722ab281bcff9270603e49621de48a106cff8dceb0135"}
Jul 17 01:07:32.660335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:32.660197    2573 scope.go:117] "RemoveContainer" containerID="9accccc6e1c84e998f0722ab281bcff9270603e49621de48a106cff8dceb0135"
Jul 17 01:07:33.669035 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:33.668970    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a57e2e-3399-424f-b463-62d4fc851e27" containerID="ffe0495e32429e957fce5eb3fa9d06b62aba591d3b5aa0a4a33e713885049e74" exitCode=6
Jul 17 01:07:33.669354 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:33.669052    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" event={"ID":"27a57e2e-3399-424f-b463-62d4fc851e27","Type":"ContainerDied","Data":"ffe0495e32429e957fce5eb3fa9d06b62aba591d3b5aa0a4a33e713885049e74"}
Jul 17 01:07:33.669354 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:33.669108    2573 scope.go:117] "RemoveContainer" containerID="9accccc6e1c84e998f0722ab281bcff9270603e49621de48a106cff8dceb0135"
Jul 17 01:07:33.669354 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:33.669282    2573 scope.go:117] "RemoveContainer" containerID="ffe0495e32429e957fce5eb3fa9d06b62aba591d3b5aa0a4a33e713885049e74"
Jul 17 01:07:33.669483 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:33.669464    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj_odh-ai-gateway-infra(27a57e2e-3399-424f-b463-62d4fc851e27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" podUID="27a57e2e-3399-424f-b463-62d4fc851e27"
Jul 17 01:07:34.676172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:34.676143    2573 scope.go:117] "RemoveContainer" containerID="ffe0495e32429e957fce5eb3fa9d06b62aba591d3b5aa0a4a33e713885049e74"
Jul 17 01:07:34.676531 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:34.676348    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj_odh-ai-gateway-infra(27a57e2e-3399-424f-b463-62d4fc851e27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" podUID="27a57e2e-3399-424f-b463-62d4fc851e27"
Jul 17 01:07:39.235409 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.235331    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"]
Jul 17 01:07:39.235808 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.235730    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:39.235808 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.235744    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d20dbcf-9612-4631-84e6-194ca284d04a" containerName="revoke-keys"
Jul 17 01:07:39.235808 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.235755    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:39.235808 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.235761    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:39.235961 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.235854    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="972b718d-6cb4-4e0c-8f95-dcf140870d57" containerName="revoke-keys"
Jul 17 01:07:39.235961 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.235863    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dcb467-6931-44c6-b0fb-e359f690ebf3" containerName="revoke-keys"
Jul 17 01:07:39.240303 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.240275    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"
Jul 17 01:07:39.245145 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.245118    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"]
Jul 17 01:07:39.366068 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.366043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac66ee-451a-4606-a6be-1351edcfbf1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd\" (UID: \"80ac66ee-451a-4606-a6be-1351edcfbf1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"
Jul 17 01:07:39.467389 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.467361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac66ee-451a-4606-a6be-1351edcfbf1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd\" (UID: \"80ac66ee-451a-4606-a6be-1351edcfbf1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"
Jul 17 01:07:39.467978 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.467957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac66ee-451a-4606-a6be-1351edcfbf1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd\" (UID: \"80ac66ee-451a-4606-a6be-1351edcfbf1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"
Jul 17 01:07:39.553875 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.553843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"
Jul 17 01:07:39.696060 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.696030    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"]
Jul 17 01:07:39.696829 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:07:39.696775    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ac66ee_451a_4606_a6be_1351edcfbf1f.slice/crio-d5b439dc33aa9b120bf41748df88a5cd5abfab633e3d7ae1c23a65caed256c99 WatchSource:0}: Error finding container d5b439dc33aa9b120bf41748df88a5cd5abfab633e3d7ae1c23a65caed256c99: Status 404 returned error can't find the container with id d5b439dc33aa9b120bf41748df88a5cd5abfab633e3d7ae1c23a65caed256c99
Jul 17 01:07:39.706211 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:39.706183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" event={"ID":"80ac66ee-451a-4606-a6be-1351edcfbf1f","Type":"ContainerStarted","Data":"d5b439dc33aa9b120bf41748df88a5cd5abfab633e3d7ae1c23a65caed256c99"}
Jul 17 01:07:40.713485 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:40.713405    2573 generic.go:356] "Generic (PLEG): container finished" podID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerID="bbd135868b3e88ff316892026a69c9f9fd34f14a070ec497b378d33440c1b7a5" exitCode=6
Jul 17 01:07:40.713485 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:40.713455    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" event={"ID":"80ac66ee-451a-4606-a6be-1351edcfbf1f","Type":"ContainerDied","Data":"bbd135868b3e88ff316892026a69c9f9fd34f14a070ec497b378d33440c1b7a5"}
Jul 17 01:07:40.713881 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:40.713676    2573 scope.go:117] "RemoveContainer" containerID="bbd135868b3e88ff316892026a69c9f9fd34f14a070ec497b378d33440c1b7a5"
Jul 17 01:07:41.720979 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:41.720905    2573 generic.go:356] "Generic (PLEG): container finished" podID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerID="b885d68b134ed9f77d84d75dcb24ea7a8bfb4dbcc40d800929617a92926b83ff" exitCode=6
Jul 17 01:07:41.721329 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:41.720990    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" event={"ID":"80ac66ee-451a-4606-a6be-1351edcfbf1f","Type":"ContainerDied","Data":"b885d68b134ed9f77d84d75dcb24ea7a8bfb4dbcc40d800929617a92926b83ff"}
Jul 17 01:07:41.721329 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:41.721061    2573 scope.go:117] "RemoveContainer" containerID="bbd135868b3e88ff316892026a69c9f9fd34f14a070ec497b378d33440c1b7a5"
Jul 17 01:07:41.721329 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:41.721241    2573 scope.go:117] "RemoveContainer" containerID="b885d68b134ed9f77d84d75dcb24ea7a8bfb4dbcc40d800929617a92926b83ff"
Jul 17 01:07:41.721494 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:41.721412    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd_odh-ai-gateway-infra(80ac66ee-451a-4606-a6be-1351edcfbf1f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f"
Jul 17 01:07:42.728765 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:42.728738    2573 scope.go:117] "RemoveContainer" containerID="b885d68b134ed9f77d84d75dcb24ea7a8bfb4dbcc40d800929617a92926b83ff"
Jul 17 01:07:42.729147 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:42.728936    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd_odh-ai-gateway-infra(80ac66ee-451a-4606-a6be-1351edcfbf1f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f"
Jul 17 01:07:43.523078 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.523044    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"]
Jul 17 01:07:43.528012 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.527992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"
Jul 17 01:07:43.532087 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.532053    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"]
Jul 17 01:07:43.601899 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.601859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef7e76-9dd1-4cc9-bc5b-b7023c975719-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d\" (UID: \"6eef7e76-9dd1-4cc9-bc5b-b7023c975719\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"
Jul 17 01:07:43.702254 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.702228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef7e76-9dd1-4cc9-bc5b-b7023c975719-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d\" (UID: \"6eef7e76-9dd1-4cc9-bc5b-b7023c975719\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"
Jul 17 01:07:43.702812 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.702773    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef7e76-9dd1-4cc9-bc5b-b7023c975719-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d\" (UID: \"6eef7e76-9dd1-4cc9-bc5b-b7023c975719\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"
Jul 17 01:07:43.841857 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.841779    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"
Jul 17 01:07:43.971714 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:07:43.971682    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eef7e76_9dd1_4cc9_bc5b_b7023c975719.slice/crio-460ec1c57cba94da445ad6b55bbcab93e1baa1edfc3a8fe7121a0679df8329f6 WatchSource:0}: Error finding container 460ec1c57cba94da445ad6b55bbcab93e1baa1edfc3a8fe7121a0679df8329f6: Status 404 returned error can't find the container with id 460ec1c57cba94da445ad6b55bbcab93e1baa1edfc3a8fe7121a0679df8329f6
Jul 17 01:07:43.972244 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:43.972221    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"]
Jul 17 01:07:44.741540 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:44.741508    2573 generic.go:356] "Generic (PLEG): container finished" podID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerID="0f5380317c9ac8b7b51494fcd8065d2bead358e4681149ae91cde8edca6c8a54" exitCode=6
Jul 17 01:07:44.741691 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:44.741584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" event={"ID":"6eef7e76-9dd1-4cc9-bc5b-b7023c975719","Type":"ContainerDied","Data":"0f5380317c9ac8b7b51494fcd8065d2bead358e4681149ae91cde8edca6c8a54"}
Jul 17 01:07:44.741691 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:44.741632    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" event={"ID":"6eef7e76-9dd1-4cc9-bc5b-b7023c975719","Type":"ContainerStarted","Data":"460ec1c57cba94da445ad6b55bbcab93e1baa1edfc3a8fe7121a0679df8329f6"}
Jul 17 01:07:44.741853 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:44.741713    2573 scope.go:117] "RemoveContainer" containerID="0f5380317c9ac8b7b51494fcd8065d2bead358e4681149ae91cde8edca6c8a54"
Jul 17 01:07:45.459566 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.459532    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rpldh"]
Jul 17 01:07:45.463100 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.463078    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:45.469183 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.469155    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rpldh"]
Jul 17 01:07:45.517010 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.516983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w44\" (UniqueName: \"kubernetes.io/projected/05ff5f33-df10-4d77-97e7-cd4409c4f8ae-kube-api-access-k6w44\") pod \"kuadrant-operator-catalog-rpldh\" (UID: \"05ff5f33-df10-4d77-97e7-cd4409c4f8ae\") " pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:45.617844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.617813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w44\" (UniqueName: \"kubernetes.io/projected/05ff5f33-df10-4d77-97e7-cd4409c4f8ae-kube-api-access-k6w44\") pod \"kuadrant-operator-catalog-rpldh\" (UID: \"05ff5f33-df10-4d77-97e7-cd4409c4f8ae\") " pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:45.625126 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.625107    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w44\" (UniqueName: \"kubernetes.io/projected/05ff5f33-df10-4d77-97e7-cd4409c4f8ae-kube-api-access-k6w44\") pod \"kuadrant-operator-catalog-rpldh\" (UID: \"05ff5f33-df10-4d77-97e7-cd4409c4f8ae\") " pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:45.749987 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.749926    2573 generic.go:356] "Generic (PLEG): container finished" podID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerID="fa3b764243dd466e661a28cb70da7189b89e18702d356445a8b36843c4841d8a" exitCode=6
Jul 17 01:07:45.750083 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.750014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" event={"ID":"6eef7e76-9dd1-4cc9-bc5b-b7023c975719","Type":"ContainerDied","Data":"fa3b764243dd466e661a28cb70da7189b89e18702d356445a8b36843c4841d8a"}
Jul 17 01:07:45.750083 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.750064    2573 scope.go:117] "RemoveContainer" containerID="0f5380317c9ac8b7b51494fcd8065d2bead358e4681149ae91cde8edca6c8a54"
Jul 17 01:07:45.750166 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.750150    2573 scope.go:117] "RemoveContainer" containerID="fa3b764243dd466e661a28cb70da7189b89e18702d356445a8b36843c4841d8a"
Jul 17 01:07:45.750330 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:45.750312    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d_odh-ai-gateway-infra(6eef7e76-9dd1-4cc9-bc5b-b7023c975719)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719"
Jul 17 01:07:45.775224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.775206    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:45.902945 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:45.902920    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rpldh"]
Jul 17 01:07:45.904250 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:07:45.904223    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ff5f33_df10_4d77_97e7_cd4409c4f8ae.slice/crio-4306ff88e36c506b4d6206486e7d288d23944ee6e4e20f8065266082486e0aa3 WatchSource:0}: Error finding container 4306ff88e36c506b4d6206486e7d288d23944ee6e4e20f8065266082486e0aa3: Status 404 returned error can't find the container with id 4306ff88e36c506b4d6206486e7d288d23944ee6e4e20f8065266082486e0aa3
Jul 17 01:07:46.757750 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:46.757720    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rpldh" event={"ID":"05ff5f33-df10-4d77-97e7-cd4409c4f8ae","Type":"ContainerStarted","Data":"96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5"}
Jul 17 01:07:46.758201 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:46.757779    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rpldh" event={"ID":"05ff5f33-df10-4d77-97e7-cd4409c4f8ae","Type":"ContainerStarted","Data":"4306ff88e36c506b4d6206486e7d288d23944ee6e4e20f8065266082486e0aa3"}
Jul 17 01:07:46.759537 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:46.759519    2573 scope.go:117] "RemoveContainer" containerID="fa3b764243dd466e661a28cb70da7189b89e18702d356445a8b36843c4841d8a"
Jul 17 01:07:46.759717 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:46.759697    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d_odh-ai-gateway-infra(6eef7e76-9dd1-4cc9-bc5b-b7023c975719)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719"
Jul 17 01:07:46.774475 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:46.774437    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rpldh" podStartSLOduration=0.999029015 podStartE2EDuration="1.774424259s" totalImagesPullingTime="775.395244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:07:46.770439339 +0000 UTC m=+3061.346591912" watchObservedRunningTime="2026-07-17 01:07:46.774424259 +0000 UTC m=+3061.350576833"
Jul 17 01:07:48.962938 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:48.962908    2573 scope.go:117] "RemoveContainer" containerID="ffe0495e32429e957fce5eb3fa9d06b62aba591d3b5aa0a4a33e713885049e74"
Jul 17 01:07:49.778496 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:49.778462    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a57e2e-3399-424f-b463-62d4fc851e27" containerID="7d054f894af8c0601f26924b65abd6c5bf1367bdc38c328c15a358023ee608ab" exitCode=6
Jul 17 01:07:49.778685 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:49.778537    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" event={"ID":"27a57e2e-3399-424f-b463-62d4fc851e27","Type":"ContainerDied","Data":"7d054f894af8c0601f26924b65abd6c5bf1367bdc38c328c15a358023ee608ab"}
Jul 17 01:07:49.778685 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:49.778601    2573 scope.go:117] "RemoveContainer" containerID="ffe0495e32429e957fce5eb3fa9d06b62aba591d3b5aa0a4a33e713885049e74"
Jul 17 01:07:49.778898 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:49.778875    2573 scope.go:117] "RemoveContainer" containerID="7d054f894af8c0601f26924b65abd6c5bf1367bdc38c328c15a358023ee608ab"
Jul 17 01:07:49.779063 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:49.779045    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj_odh-ai-gateway-infra(27a57e2e-3399-424f-b463-62d4fc851e27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" podUID="27a57e2e-3399-424f-b463-62d4fc851e27"
Jul 17 01:07:50.801208 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:50.801172    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"]
Jul 17 01:07:50.949437 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:50.949414    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"
Jul 17 01:07:51.062433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.062346    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a57e2e-3399-424f-b463-62d4fc851e27-maas-api-service-ca\") pod \"27a57e2e-3399-424f-b463-62d4fc851e27\" (UID: \"27a57e2e-3399-424f-b463-62d4fc851e27\") "
Jul 17 01:07:51.062748 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.062720    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a57e2e-3399-424f-b463-62d4fc851e27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27a57e2e-3399-424f-b463-62d4fc851e27" (UID: "27a57e2e-3399-424f-b463-62d4fc851e27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:07:51.163084 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.163059    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a57e2e-3399-424f-b463-62d4fc851e27-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:07:51.793969 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.793934    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"
Jul 17 01:07:51.794161 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.793960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj" event={"ID":"27a57e2e-3399-424f-b463-62d4fc851e27","Type":"ContainerDied","Data":"b4c454d269337d9107a765377449823275922e8fdfdcd79aca6111cb3ede8ca8"}
Jul 17 01:07:51.794161 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.794053    2573 scope.go:117] "RemoveContainer" containerID="7d054f894af8c0601f26924b65abd6c5bf1367bdc38c328c15a358023ee608ab"
Jul 17 01:07:51.823164 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.823110    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"]
Jul 17 01:07:51.827415 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.827384    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j74mj"]
Jul 17 01:07:51.968567 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:51.968542    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" path="/var/lib/kubelet/pods/27a57e2e-3399-424f-b463-62d4fc851e27/volumes"
Jul 17 01:07:53.963278 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:53.963245    2573 scope.go:117] "RemoveContainer" containerID="b885d68b134ed9f77d84d75dcb24ea7a8bfb4dbcc40d800929617a92926b83ff"
Jul 17 01:07:54.814153 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:54.814124    2573 generic.go:356] "Generic (PLEG): container finished" podID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerID="4581dda281bd7271663c437a9c03013aafbc69e187c42babb9343e31d3f6c528" exitCode=6
Jul 17 01:07:54.814288 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:54.814201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" event={"ID":"80ac66ee-451a-4606-a6be-1351edcfbf1f","Type":"ContainerDied","Data":"4581dda281bd7271663c437a9c03013aafbc69e187c42babb9343e31d3f6c528"}
Jul 17 01:07:54.814288 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:54.814262    2573 scope.go:117] "RemoveContainer" containerID="b885d68b134ed9f77d84d75dcb24ea7a8bfb4dbcc40d800929617a92926b83ff"
Jul 17 01:07:54.814573 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:54.814554    2573 scope.go:117] "RemoveContainer" containerID="4581dda281bd7271663c437a9c03013aafbc69e187c42babb9343e31d3f6c528"
Jul 17 01:07:54.814760 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:54.814738    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd_odh-ai-gateway-infra(80ac66ee-451a-4606-a6be-1351edcfbf1f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f"
Jul 17 01:07:55.776213 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:55.776179    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:55.776615 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:55.776242    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:55.798444 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:55.798421    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:55.838316 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:55.838284    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"]
Jul 17 01:07:55.844177 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:55.844156    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:55.976851 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:55.976825    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"
Jul 17 01:07:56.022631 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.022603    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rpldh"]
Jul 17 01:07:56.102535 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.102467    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac66ee-451a-4606-a6be-1351edcfbf1f-maas-api-service-ca\") pod \"80ac66ee-451a-4606-a6be-1351edcfbf1f\" (UID: \"80ac66ee-451a-4606-a6be-1351edcfbf1f\") "
Jul 17 01:07:56.102767 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.102746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ac66ee-451a-4606-a6be-1351edcfbf1f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80ac66ee-451a-4606-a6be-1351edcfbf1f" (UID: "80ac66ee-451a-4606-a6be-1351edcfbf1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:07:56.203683 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.203657    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac66ee-451a-4606-a6be-1351edcfbf1f-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:07:56.559800 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.559756    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"]
Jul 17 01:07:56.560155 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560142    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:07:56.560200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560156    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:07:56.560200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560167    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:07:56.560200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560173    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:07:56.560200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560180    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:07:56.560200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560186    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:07:56.560200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560193    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:07:56.560200 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560198    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:07:56.560412 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560260    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:07:56.560412 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560267    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:07:56.560412 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560274    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:07:56.560412 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560279    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:07:56.560412 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.560284    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:07:56.562462 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.562445    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"
Jul 17 01:07:56.568338 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.568285    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"]
Jul 17 01:07:56.708700 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.708667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4099b9b1-d7e8-4663-b293-e9cc069bce0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-g9xmp\" (UID: \"4099b9b1-d7e8-4663-b293-e9cc069bce0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"
Jul 17 01:07:56.809505 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.809479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4099b9b1-d7e8-4663-b293-e9cc069bce0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-g9xmp\" (UID: \"4099b9b1-d7e8-4663-b293-e9cc069bce0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"
Jul 17 01:07:56.810240 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.810186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4099b9b1-d7e8-4663-b293-e9cc069bce0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-g9xmp\" (UID: \"4099b9b1-d7e8-4663-b293-e9cc069bce0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"
Jul 17 01:07:56.827306 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.827286    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"
Jul 17 01:07:56.827415 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.827335    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd" event={"ID":"80ac66ee-451a-4606-a6be-1351edcfbf1f","Type":"ContainerDied","Data":"d5b439dc33aa9b120bf41748df88a5cd5abfab633e3d7ae1c23a65caed256c99"}
Jul 17 01:07:56.827415 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.827391    2573 scope.go:117] "RemoveContainer" containerID="4581dda281bd7271663c437a9c03013aafbc69e187c42babb9343e31d3f6c528"
Jul 17 01:07:56.850409 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.850384    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"]
Jul 17 01:07:56.858033 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.858010    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-pwxmd"]
Jul 17 01:07:56.875739 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:56.875718    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"
Jul 17 01:07:57.010806 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.010761    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"]
Jul 17 01:07:57.013321 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:07:57.013293    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4099b9b1_d7e8_4663_b293_e9cc069bce0e.slice/crio-57d9d27ee53fa183fabbc860d779a82ff965eb4d74966abde7c95c9692c5dce8 WatchSource:0}: Error finding container 57d9d27ee53fa183fabbc860d779a82ff965eb4d74966abde7c95c9692c5dce8: Status 404 returned error can't find the container with id 57d9d27ee53fa183fabbc860d779a82ff965eb4d74966abde7c95c9692c5dce8
Jul 17 01:07:57.834904 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.834875    2573 generic.go:356] "Generic (PLEG): container finished" podID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerID="ada27112ada230b08f006f375c11a0fba0e6522a4fb45302063b1791341bd798" exitCode=6
Jul 17 01:07:57.835277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.834966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" event={"ID":"4099b9b1-d7e8-4663-b293-e9cc069bce0e","Type":"ContainerDied","Data":"ada27112ada230b08f006f375c11a0fba0e6522a4fb45302063b1791341bd798"}
Jul 17 01:07:57.835277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.835032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" event={"ID":"4099b9b1-d7e8-4663-b293-e9cc069bce0e","Type":"ContainerStarted","Data":"57d9d27ee53fa183fabbc860d779a82ff965eb4d74966abde7c95c9692c5dce8"}
Jul 17 01:07:57.835277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.835059    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-rpldh" podUID="05ff5f33-df10-4d77-97e7-cd4409c4f8ae" containerName="registry-server" containerID="cri-o://96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5" gracePeriod=2
Jul 17 01:07:57.835277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.835081    2573 scope.go:117] "RemoveContainer" containerID="ada27112ada230b08f006f375c11a0fba0e6522a4fb45302063b1791341bd798"
Jul 17 01:07:57.962916 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.962896    2573 scope.go:117] "RemoveContainer" containerID="fa3b764243dd466e661a28cb70da7189b89e18702d356445a8b36843c4841d8a"
Jul 17 01:07:57.970271 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:57.970246    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" path="/var/lib/kubelet/pods/80ac66ee-451a-4606-a6be-1351edcfbf1f/volumes"
Jul 17 01:07:58.178085 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.178056    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:58.322858 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.322833    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6w44\" (UniqueName: \"kubernetes.io/projected/05ff5f33-df10-4d77-97e7-cd4409c4f8ae-kube-api-access-k6w44\") pod \"05ff5f33-df10-4d77-97e7-cd4409c4f8ae\" (UID: \"05ff5f33-df10-4d77-97e7-cd4409c4f8ae\") "
Jul 17 01:07:58.324704 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.324682    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ff5f33-df10-4d77-97e7-cd4409c4f8ae-kube-api-access-k6w44" (OuterVolumeSpecName: "kube-api-access-k6w44") pod "05ff5f33-df10-4d77-97e7-cd4409c4f8ae" (UID: "05ff5f33-df10-4d77-97e7-cd4409c4f8ae"). InnerVolumeSpecName "kube-api-access-k6w44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:07:58.423542 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.423521    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6w44\" (UniqueName: \"kubernetes.io/projected/05ff5f33-df10-4d77-97e7-cd4409c4f8ae-kube-api-access-k6w44\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:07:58.849427 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.849394    2573 generic.go:356] "Generic (PLEG): container finished" podID="05ff5f33-df10-4d77-97e7-cd4409c4f8ae" containerID="96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5" exitCode=0
Jul 17 01:07:58.849904 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.849435    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rpldh"
Jul 17 01:07:58.849904 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.849468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rpldh" event={"ID":"05ff5f33-df10-4d77-97e7-cd4409c4f8ae","Type":"ContainerDied","Data":"96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5"}
Jul 17 01:07:58.849904 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.849558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rpldh" event={"ID":"05ff5f33-df10-4d77-97e7-cd4409c4f8ae","Type":"ContainerDied","Data":"4306ff88e36c506b4d6206486e7d288d23944ee6e4e20f8065266082486e0aa3"}
Jul 17 01:07:58.849904 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.849588    2573 scope.go:117] "RemoveContainer" containerID="96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5"
Jul 17 01:07:58.851603 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.851580    2573 generic.go:356] "Generic (PLEG): container finished" podID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerID="14e400e6390a3952b2e22c55cc03b7e75a03af44fceca4bbeffa8e4177c09dff" exitCode=6
Jul 17 01:07:58.851700 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.851658    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" event={"ID":"6eef7e76-9dd1-4cc9-bc5b-b7023c975719","Type":"ContainerDied","Data":"14e400e6390a3952b2e22c55cc03b7e75a03af44fceca4bbeffa8e4177c09dff"}
Jul 17 01:07:58.852019 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.852000    2573 scope.go:117] "RemoveContainer" containerID="14e400e6390a3952b2e22c55cc03b7e75a03af44fceca4bbeffa8e4177c09dff"
Jul 17 01:07:58.852227 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:58.852205    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d_odh-ai-gateway-infra(6eef7e76-9dd1-4cc9-bc5b-b7023c975719)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719"
Jul 17 01:07:58.853701 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.853678    2573 generic.go:356] "Generic (PLEG): container finished" podID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerID="13be2f3d7b51574cf97a8baa61c33229aea3a3a1150bf88ecc1f2aca40a4fa02" exitCode=6
Jul 17 01:07:58.853836 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.853750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" event={"ID":"4099b9b1-d7e8-4663-b293-e9cc069bce0e","Type":"ContainerDied","Data":"13be2f3d7b51574cf97a8baa61c33229aea3a3a1150bf88ecc1f2aca40a4fa02"}
Jul 17 01:07:58.854002 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.853987    2573 scope.go:117] "RemoveContainer" containerID="13be2f3d7b51574cf97a8baa61c33229aea3a3a1150bf88ecc1f2aca40a4fa02"
Jul 17 01:07:58.854191 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:58.854171    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-g9xmp_odh-ai-gateway-infra(4099b9b1-d7e8-4663-b293-e9cc069bce0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e"
Jul 17 01:07:58.860345 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.860327    2573 scope.go:117] "RemoveContainer" containerID="96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5"
Jul 17 01:07:58.860576 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:58.860559    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5\": container with ID starting with 96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5 not found: ID does not exist" containerID="96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5"
Jul 17 01:07:58.860639 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.860583    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5"} err="failed to get container status \"96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5\": rpc error: code = NotFound desc = could not find container \"96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5\": container with ID starting with 96327666b9c2c2ba272c72c3e18864cc7735c444a82250a4d155d4cb018f84d5 not found: ID does not exist"
Jul 17 01:07:58.860639 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.860599    2573 scope.go:117] "RemoveContainer" containerID="fa3b764243dd466e661a28cb70da7189b89e18702d356445a8b36843c4841d8a"
Jul 17 01:07:58.871143 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.871122    2573 scope.go:117] "RemoveContainer" containerID="ada27112ada230b08f006f375c11a0fba0e6522a4fb45302063b1791341bd798"
Jul 17 01:07:58.890352 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.890332    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rpldh"]
Jul 17 01:07:58.893636 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:58.893588    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rpldh"]
Jul 17 01:07:59.862842 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:59.862817    2573 scope.go:117] "RemoveContainer" containerID="13be2f3d7b51574cf97a8baa61c33229aea3a3a1150bf88ecc1f2aca40a4fa02"
Jul 17 01:07:59.863218 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:07:59.863001    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-g9xmp_odh-ai-gateway-infra(4099b9b1-d7e8-4663-b293-e9cc069bce0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e"
Jul 17 01:07:59.877046 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:59.876858    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"]
Jul 17 01:07:59.968850 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:07:59.968819    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ff5f33-df10-4d77-97e7-cd4409c4f8ae" path="/var/lib/kubelet/pods/05ff5f33-df10-4d77-97e7-cd4409c4f8ae/volumes"
Jul 17 01:08:00.019880 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.019858    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"
Jul 17 01:08:00.138537 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.138455    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef7e76-9dd1-4cc9-bc5b-b7023c975719-maas-api-service-ca\") pod \"6eef7e76-9dd1-4cc9-bc5b-b7023c975719\" (UID: \"6eef7e76-9dd1-4cc9-bc5b-b7023c975719\") "
Jul 17 01:08:00.138883 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.138863    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eef7e76-9dd1-4cc9-bc5b-b7023c975719-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6eef7e76-9dd1-4cc9-bc5b-b7023c975719" (UID: "6eef7e76-9dd1-4cc9-bc5b-b7023c975719"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:08:00.239360 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.239329    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef7e76-9dd1-4cc9-bc5b-b7023c975719-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:08:00.870026 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.869999    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"
Jul 17 01:08:00.870457 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.870029    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d" event={"ID":"6eef7e76-9dd1-4cc9-bc5b-b7023c975719","Type":"ContainerDied","Data":"460ec1c57cba94da445ad6b55bbcab93e1baa1edfc3a8fe7121a0679df8329f6"}
Jul 17 01:08:00.870457 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.870094    2573 scope.go:117] "RemoveContainer" containerID="14e400e6390a3952b2e22c55cc03b7e75a03af44fceca4bbeffa8e4177c09dff"
Jul 17 01:08:00.893948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.893926    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"]
Jul 17 01:08:00.896320 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:00.896301    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-vnv6d"]
Jul 17 01:08:01.968515 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:01.968483    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" path="/var/lib/kubelet/pods/6eef7e76-9dd1-4cc9-bc5b-b7023c975719/volumes"
Jul 17 01:08:12.962388 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:12.962361    2573 scope.go:117] "RemoveContainer" containerID="13be2f3d7b51574cf97a8baa61c33229aea3a3a1150bf88ecc1f2aca40a4fa02"
Jul 17 01:08:13.953687 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:13.953604    2573 generic.go:356] "Generic (PLEG): container finished" podID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerID="2c807502578da00943c775ed5c92da54c5e1f296c5f04a8543854be07d039d87" exitCode=6
Jul 17 01:08:13.953687 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:13.953676    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" event={"ID":"4099b9b1-d7e8-4663-b293-e9cc069bce0e","Type":"ContainerDied","Data":"2c807502578da00943c775ed5c92da54c5e1f296c5f04a8543854be07d039d87"}
Jul 17 01:08:13.953956 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:13.953737    2573 scope.go:117] "RemoveContainer" containerID="13be2f3d7b51574cf97a8baa61c33229aea3a3a1150bf88ecc1f2aca40a4fa02"
Jul 17 01:08:13.954073 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:13.954056    2573 scope.go:117] "RemoveContainer" containerID="2c807502578da00943c775ed5c92da54c5e1f296c5f04a8543854be07d039d87"
Jul 17 01:08:13.954265 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:13.954245    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-g9xmp_odh-ai-gateway-infra(4099b9b1-d7e8-4663-b293-e9cc069bce0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e"
Jul 17 01:08:14.977250 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:14.977218    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"]
Jul 17 01:08:15.112957 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.112933    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"
Jul 17 01:08:15.262812 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.262762    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4099b9b1-d7e8-4663-b293-e9cc069bce0e-maas-api-service-ca\") pod \"4099b9b1-d7e8-4663-b293-e9cc069bce0e\" (UID: \"4099b9b1-d7e8-4663-b293-e9cc069bce0e\") "
Jul 17 01:08:15.263089 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.263068    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4099b9b1-d7e8-4663-b293-e9cc069bce0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4099b9b1-d7e8-4663-b293-e9cc069bce0e" (UID: "4099b9b1-d7e8-4663-b293-e9cc069bce0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:08:15.364095 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.364068    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4099b9b1-d7e8-4663-b293-e9cc069bce0e-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:08:15.975661 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.975625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp" event={"ID":"4099b9b1-d7e8-4663-b293-e9cc069bce0e","Type":"ContainerDied","Data":"57d9d27ee53fa183fabbc860d779a82ff965eb4d74966abde7c95c9692c5dce8"}
Jul 17 01:08:15.975844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.975667    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"
Jul 17 01:08:15.975844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.975685    2573 scope.go:117] "RemoveContainer" containerID="2c807502578da00943c775ed5c92da54c5e1f296c5f04a8543854be07d039d87"
Jul 17 01:08:15.997610 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:15.997582    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"]
Jul 17 01:08:16.001027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:16.001003    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-g9xmp"]
Jul 17 01:08:17.969933 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:17.969893    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" path="/var/lib/kubelet/pods/4099b9b1-d7e8-4663-b293-e9cc069bce0e/volumes"
Jul 17 01:08:29.290132 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290097    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"]
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290465    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290477    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290489    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ff5f33-df10-4d77-97e7-cd4409c4f8ae" containerName="registry-server"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290494    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ff5f33-df10-4d77-97e7-cd4409c4f8ae" containerName="registry-server"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290510    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290517    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290526    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290531    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290538    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290543    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290549    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290554    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a57e2e-3399-424f-b463-62d4fc851e27" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290559    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290564    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290570    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:29.290597 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290574    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:29.291172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290625    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:29.291172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290632    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.291172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290637    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ac66ee-451a-4606-a6be-1351edcfbf1f" containerName="revoke-keys"
Jul 17 01:08:29.291172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290644    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ff5f33-df10-4d77-97e7-cd4409c4f8ae" containerName="registry-server"
Jul 17 01:08:29.291172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290649    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:29.291172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.290655    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:29.293566 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.293549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"
Jul 17 01:08:29.295456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.295438    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:08:29.300535 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.300510    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"]
Jul 17 01:08:29.378524 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.378483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b143fb-4421-4d87-98e2-dc747076ec8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8\" (UID: \"c1b143fb-4421-4d87-98e2-dc747076ec8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"
Jul 17 01:08:29.479091 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.479067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b143fb-4421-4d87-98e2-dc747076ec8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8\" (UID: \"c1b143fb-4421-4d87-98e2-dc747076ec8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"
Jul 17 01:08:29.479661 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.479642    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b143fb-4421-4d87-98e2-dc747076ec8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8\" (UID: \"c1b143fb-4421-4d87-98e2-dc747076ec8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"
Jul 17 01:08:29.604938 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.604876    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"
Jul 17 01:08:29.734203 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:29.734167    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"]
Jul 17 01:08:29.734973 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:08:29.734946    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b143fb_4421_4d87_98e2_dc747076ec8d.slice/crio-235beb2935aa18f8fac2e394689e0e0d42363f194465055d52f10628b4d4e34f WatchSource:0}: Error finding container 235beb2935aa18f8fac2e394689e0e0d42363f194465055d52f10628b4d4e34f: Status 404 returned error can't find the container with id 235beb2935aa18f8fac2e394689e0e0d42363f194465055d52f10628b4d4e34f
Jul 17 01:08:30.055283 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:30.055257    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" event={"ID":"c1b143fb-4421-4d87-98e2-dc747076ec8d","Type":"ContainerStarted","Data":"72a951b0162ffb0d6cb2c7a4970640314a5474e8e47187bea85bb5532e8bc07d"}
Jul 17 01:08:30.055500 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:30.055305    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" event={"ID":"c1b143fb-4421-4d87-98e2-dc747076ec8d","Type":"ContainerStarted","Data":"235beb2935aa18f8fac2e394689e0e0d42363f194465055d52f10628b4d4e34f"}
Jul 17 01:08:30.069324 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:30.069281    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" podStartSLOduration=1.06926623 podStartE2EDuration="1.06926623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:08:30.068009115 +0000 UTC m=+3104.644161691" watchObservedRunningTime="2026-07-17 01:08:30.06926623 +0000 UTC m=+3104.645418849"
Jul 17 01:08:31.062272 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.062236    2573 generic.go:356] "Generic (PLEG): container finished" podID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerID="72a951b0162ffb0d6cb2c7a4970640314a5474e8e47187bea85bb5532e8bc07d" exitCode=6
Jul 17 01:08:31.062272 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.062274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" event={"ID":"c1b143fb-4421-4d87-98e2-dc747076ec8d","Type":"ContainerDied","Data":"72a951b0162ffb0d6cb2c7a4970640314a5474e8e47187bea85bb5532e8bc07d"}
Jul 17 01:08:31.062835 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.062542    2573 scope.go:117] "RemoveContainer" containerID="72a951b0162ffb0d6cb2c7a4970640314a5474e8e47187bea85bb5532e8bc07d"
Jul 17 01:08:31.135162 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.135134    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"]
Jul 17 01:08:31.135516 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.135505    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:31.135560 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.135519    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:31.135596 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.135586    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4099b9b1-d7e8-4663-b293-e9cc069bce0e" containerName="revoke-keys"
Jul 17 01:08:31.135596 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.135594    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eef7e76-9dd1-4cc9-bc5b-b7023c975719" containerName="revoke-keys"
Jul 17 01:08:31.139606 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.139586    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"
Jul 17 01:08:31.145315 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.145291    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"]
Jul 17 01:08:31.195668 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.195627    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe0e4d8-74e6-4030-92ad-c4673cd69def-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq\" (UID: \"3fe0e4d8-74e6-4030-92ad-c4673cd69def\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"
Jul 17 01:08:31.296302 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.296270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe0e4d8-74e6-4030-92ad-c4673cd69def-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq\" (UID: \"3fe0e4d8-74e6-4030-92ad-c4673cd69def\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"
Jul 17 01:08:31.296896 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.296876    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe0e4d8-74e6-4030-92ad-c4673cd69def-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq\" (UID: \"3fe0e4d8-74e6-4030-92ad-c4673cd69def\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"
Jul 17 01:08:31.452917 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.452835    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"
Jul 17 01:08:31.593158 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:31.593133    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"]
Jul 17 01:08:31.594872 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:08:31.594845    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe0e4d8_74e6_4030_92ad_c4673cd69def.slice/crio-1b33d180d45d01250205924bd3156429c2e859fd37025c40addc61003744241d WatchSource:0}: Error finding container 1b33d180d45d01250205924bd3156429c2e859fd37025c40addc61003744241d: Status 404 returned error can't find the container with id 1b33d180d45d01250205924bd3156429c2e859fd37025c40addc61003744241d
Jul 17 01:08:32.070506 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:32.070469    2573 generic.go:356] "Generic (PLEG): container finished" podID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerID="c630a1ed0e652e66e390505ac77f3e17ac9a7ae1d76aca81a31809827cf544ef" exitCode=6
Jul 17 01:08:32.071023 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:32.070559    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" event={"ID":"c1b143fb-4421-4d87-98e2-dc747076ec8d","Type":"ContainerDied","Data":"c630a1ed0e652e66e390505ac77f3e17ac9a7ae1d76aca81a31809827cf544ef"}
Jul 17 01:08:32.071023 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:32.070627    2573 scope.go:117] "RemoveContainer" containerID="72a951b0162ffb0d6cb2c7a4970640314a5474e8e47187bea85bb5532e8bc07d"
Jul 17 01:08:32.071023 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:32.070931    2573 scope.go:117] "RemoveContainer" containerID="c630a1ed0e652e66e390505ac77f3e17ac9a7ae1d76aca81a31809827cf544ef"
Jul 17 01:08:32.071219 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:32.071187    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8_odh-ai-gateway-infra(c1b143fb-4421-4d87-98e2-dc747076ec8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d"
Jul 17 01:08:32.072496 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:32.072467    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" event={"ID":"3fe0e4d8-74e6-4030-92ad-c4673cd69def","Type":"ContainerStarted","Data":"6ec1da139f555e059570bb8e92c4aa3c2175d688c6d679f91ebcd5c7e375d876"}
Jul 17 01:08:32.072576 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:32.072520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" event={"ID":"3fe0e4d8-74e6-4030-92ad-c4673cd69def","Type":"ContainerStarted","Data":"1b33d180d45d01250205924bd3156429c2e859fd37025c40addc61003744241d"}
Jul 17 01:08:32.097500 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:32.097446    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" podStartSLOduration=1.097430051 podStartE2EDuration="1.097430051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:08:32.094176764 +0000 UTC m=+3106.670329329" watchObservedRunningTime="2026-07-17 01:08:32.097430051 +0000 UTC m=+3106.673582624"
Jul 17 01:08:33.088321 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.088289    2573 scope.go:117] "RemoveContainer" containerID="c630a1ed0e652e66e390505ac77f3e17ac9a7ae1d76aca81a31809827cf544ef"
Jul 17 01:08:33.088743 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:33.088541    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8_odh-ai-gateway-infra(c1b143fb-4421-4d87-98e2-dc747076ec8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d"
Jul 17 01:08:33.089953 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.089927    2573 generic.go:356] "Generic (PLEG): container finished" podID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerID="6ec1da139f555e059570bb8e92c4aa3c2175d688c6d679f91ebcd5c7e375d876" exitCode=6
Jul 17 01:08:33.090083 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.090015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" event={"ID":"3fe0e4d8-74e6-4030-92ad-c4673cd69def","Type":"ContainerDied","Data":"6ec1da139f555e059570bb8e92c4aa3c2175d688c6d679f91ebcd5c7e375d876"}
Jul 17 01:08:33.090178 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.090163    2573 scope.go:117] "RemoveContainer" containerID="6ec1da139f555e059570bb8e92c4aa3c2175d688c6d679f91ebcd5c7e375d876"
Jul 17 01:08:33.565090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.565036    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"]
Jul 17 01:08:33.568650 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.568626    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"
Jul 17 01:08:33.574811 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.574761    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"]
Jul 17 01:08:33.618688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.618581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79288127-53ee-4047-91bf-a0488116bf84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r\" (UID: \"79288127-53ee-4047-91bf-a0488116bf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"
Jul 17 01:08:33.719768 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.719727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79288127-53ee-4047-91bf-a0488116bf84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r\" (UID: \"79288127-53ee-4047-91bf-a0488116bf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"
Jul 17 01:08:33.720379 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.720358    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79288127-53ee-4047-91bf-a0488116bf84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r\" (UID: \"79288127-53ee-4047-91bf-a0488116bf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"
Jul 17 01:08:33.882942 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:33.882837    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"
Jul 17 01:08:34.042315 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:34.042278    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"]
Jul 17 01:08:34.042848 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:08:34.042786    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79288127_53ee_4047_91bf_a0488116bf84.slice/crio-6f14c93063f4f76561a616ce7149d3161c39267a12ac548b046230dceb8bff3f WatchSource:0}: Error finding container 6f14c93063f4f76561a616ce7149d3161c39267a12ac548b046230dceb8bff3f: Status 404 returned error can't find the container with id 6f14c93063f4f76561a616ce7149d3161c39267a12ac548b046230dceb8bff3f
Jul 17 01:08:34.097688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:34.097652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" event={"ID":"79288127-53ee-4047-91bf-a0488116bf84","Type":"ContainerStarted","Data":"6f14c93063f4f76561a616ce7149d3161c39267a12ac548b046230dceb8bff3f"}
Jul 17 01:08:34.099660 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:34.099627    2573 generic.go:356] "Generic (PLEG): container finished" podID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerID="b41847bf414efc294d4c5b98f0ec2960a176478fc2a8f4a3991f34a74e6dba12" exitCode=6
Jul 17 01:08:34.099815 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:34.099708    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" event={"ID":"3fe0e4d8-74e6-4030-92ad-c4673cd69def","Type":"ContainerDied","Data":"b41847bf414efc294d4c5b98f0ec2960a176478fc2a8f4a3991f34a74e6dba12"}
Jul 17 01:08:34.099815 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:34.099767    2573 scope.go:117] "RemoveContainer" containerID="6ec1da139f555e059570bb8e92c4aa3c2175d688c6d679f91ebcd5c7e375d876"
Jul 17 01:08:34.100044 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:34.100024    2573 scope.go:117] "RemoveContainer" containerID="b41847bf414efc294d4c5b98f0ec2960a176478fc2a8f4a3991f34a74e6dba12"
Jul 17 01:08:34.100286 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:34.100263    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq_odh-ai-gateway-infra(3fe0e4d8-74e6-4030-92ad-c4673cd69def)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def"
Jul 17 01:08:35.108440 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:35.108405    2573 generic.go:356] "Generic (PLEG): container finished" podID="79288127-53ee-4047-91bf-a0488116bf84" containerID="00c1f46305f6bd1bf14bc6b879119134e163b8b096bfd5da85c11cc81608c296" exitCode=6
Jul 17 01:08:35.108991 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:35.108487    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" event={"ID":"79288127-53ee-4047-91bf-a0488116bf84","Type":"ContainerDied","Data":"00c1f46305f6bd1bf14bc6b879119134e163b8b096bfd5da85c11cc81608c296"}
Jul 17 01:08:35.108991 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:35.108731    2573 scope.go:117] "RemoveContainer" containerID="00c1f46305f6bd1bf14bc6b879119134e163b8b096bfd5da85c11cc81608c296"
Jul 17 01:08:35.110661 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:35.110641    2573 scope.go:117] "RemoveContainer" containerID="b41847bf414efc294d4c5b98f0ec2960a176478fc2a8f4a3991f34a74e6dba12"
Jul 17 01:08:35.110934 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:35.110910    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq_odh-ai-gateway-infra(3fe0e4d8-74e6-4030-92ad-c4673cd69def)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def"
Jul 17 01:08:36.119277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:36.119180    2573 generic.go:356] "Generic (PLEG): container finished" podID="79288127-53ee-4047-91bf-a0488116bf84" containerID="ac618e7b78963fc251d80531276db6075bd64c0752ce7beb080549cf3ba561da" exitCode=6
Jul 17 01:08:36.119277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:36.119231    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" event={"ID":"79288127-53ee-4047-91bf-a0488116bf84","Type":"ContainerDied","Data":"ac618e7b78963fc251d80531276db6075bd64c0752ce7beb080549cf3ba561da"}
Jul 17 01:08:36.119805 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:36.119282    2573 scope.go:117] "RemoveContainer" containerID="00c1f46305f6bd1bf14bc6b879119134e163b8b096bfd5da85c11cc81608c296"
Jul 17 01:08:36.119805 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:36.119477    2573 scope.go:117] "RemoveContainer" containerID="ac618e7b78963fc251d80531276db6075bd64c0752ce7beb080549cf3ba561da"
Jul 17 01:08:36.119805 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:36.119734    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r_odh-ai-gateway-infra(79288127-53ee-4047-91bf-a0488116bf84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" podUID="79288127-53ee-4047-91bf-a0488116bf84"
Jul 17 01:08:37.128989 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:37.128950    2573 scope.go:117] "RemoveContainer" containerID="ac618e7b78963fc251d80531276db6075bd64c0752ce7beb080549cf3ba561da"
Jul 17 01:08:37.129435 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:37.129234    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r_odh-ai-gateway-infra(79288127-53ee-4047-91bf-a0488116bf84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" podUID="79288127-53ee-4047-91bf-a0488116bf84"
Jul 17 01:08:44.962905 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:44.962862    2573 scope.go:117] "RemoveContainer" containerID="c630a1ed0e652e66e390505ac77f3e17ac9a7ae1d76aca81a31809827cf544ef"
Jul 17 01:08:46.188301 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.188261    2573 generic.go:356] "Generic (PLEG): container finished" podID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerID="9e3f0c507f539acd938ebfc43ae33b381a2808ed93179ef46224963286abf1b9" exitCode=6
Jul 17 01:08:46.188844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.188337    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" event={"ID":"c1b143fb-4421-4d87-98e2-dc747076ec8d","Type":"ContainerDied","Data":"9e3f0c507f539acd938ebfc43ae33b381a2808ed93179ef46224963286abf1b9"}
Jul 17 01:08:46.188844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.188426    2573 scope.go:117] "RemoveContainer" containerID="c630a1ed0e652e66e390505ac77f3e17ac9a7ae1d76aca81a31809827cf544ef"
Jul 17 01:08:46.188844 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.188763    2573 scope.go:117] "RemoveContainer" containerID="9e3f0c507f539acd938ebfc43ae33b381a2808ed93179ef46224963286abf1b9"
Jul 17 01:08:46.189086 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:46.189056    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8_odh-ai-gateway-infra(c1b143fb-4421-4d87-98e2-dc747076ec8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d"
Jul 17 01:08:46.600936 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.600895    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"]
Jul 17 01:08:46.606168 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.606135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"
Jul 17 01:08:46.611635 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.611597    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"]
Jul 17 01:08:46.750860 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.750774    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51301041-e846-4cfc-8f1b-a876b52d4434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-wfr86\" (UID: \"51301041-e846-4cfc-8f1b-a876b52d4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"
Jul 17 01:08:46.851753 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.851618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51301041-e846-4cfc-8f1b-a876b52d4434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-wfr86\" (UID: \"51301041-e846-4cfc-8f1b-a876b52d4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"
Jul 17 01:08:46.852471 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.852448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51301041-e846-4cfc-8f1b-a876b52d4434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-wfr86\" (UID: \"51301041-e846-4cfc-8f1b-a876b52d4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"
Jul 17 01:08:46.920589 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:46.920548    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"
Jul 17 01:08:47.084497 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.084461    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"]
Jul 17 01:08:47.086928 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:08:47.086896    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51301041_e846_4cfc_8f1b_a876b52d4434.slice/crio-e1678627e05f11f82d3a92e69449193fcca44d5423c4f593544e61fc8e7fc6ed WatchSource:0}: Error finding container e1678627e05f11f82d3a92e69449193fcca44d5423c4f593544e61fc8e7fc6ed: Status 404 returned error can't find the container with id e1678627e05f11f82d3a92e69449193fcca44d5423c4f593544e61fc8e7fc6ed
Jul 17 01:08:47.198106 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.198062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" event={"ID":"51301041-e846-4cfc-8f1b-a876b52d4434","Type":"ContainerStarted","Data":"e1678627e05f11f82d3a92e69449193fcca44d5423c4f593544e61fc8e7fc6ed"}
Jul 17 01:08:47.211454 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.211408    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"]
Jul 17 01:08:47.378010 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.377968    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"
Jul 17 01:08:47.459607 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.459570    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b143fb-4421-4d87-98e2-dc747076ec8d-maas-api-service-ca\") pod \"c1b143fb-4421-4d87-98e2-dc747076ec8d\" (UID: \"c1b143fb-4421-4d87-98e2-dc747076ec8d\") "
Jul 17 01:08:47.459944 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.459921    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b143fb-4421-4d87-98e2-dc747076ec8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1b143fb-4421-4d87-98e2-dc747076ec8d" (UID: "c1b143fb-4421-4d87-98e2-dc747076ec8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:08:47.561001 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.560955    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b143fb-4421-4d87-98e2-dc747076ec8d-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:08:47.963303 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:47.963257    2573 scope.go:117] "RemoveContainer" containerID="b41847bf414efc294d4c5b98f0ec2960a176478fc2a8f4a3991f34a74e6dba12"
Jul 17 01:08:48.205865 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.205824    2573 generic.go:356] "Generic (PLEG): container finished" podID="51301041-e846-4cfc-8f1b-a876b52d4434" containerID="f836676ae3d0ac3cf73a32cd0bfac5b23987dfd62d05f17f528d877283a4e713" exitCode=6
Jul 17 01:08:48.206325 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.205926    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" event={"ID":"51301041-e846-4cfc-8f1b-a876b52d4434","Type":"ContainerDied","Data":"f836676ae3d0ac3cf73a32cd0bfac5b23987dfd62d05f17f528d877283a4e713"}
Jul 17 01:08:48.206325 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.206123    2573 scope.go:117] "RemoveContainer" containerID="f836676ae3d0ac3cf73a32cd0bfac5b23987dfd62d05f17f528d877283a4e713"
Jul 17 01:08:48.207527 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.207507    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"
Jul 17 01:08:48.207527 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.207518    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8" event={"ID":"c1b143fb-4421-4d87-98e2-dc747076ec8d","Type":"ContainerDied","Data":"235beb2935aa18f8fac2e394689e0e0d42363f194465055d52f10628b4d4e34f"}
Jul 17 01:08:48.207670 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.207567    2573 scope.go:117] "RemoveContainer" containerID="9e3f0c507f539acd938ebfc43ae33b381a2808ed93179ef46224963286abf1b9"
Jul 17 01:08:48.233139 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.233091    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"]
Jul 17 01:08:48.239212 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.239177    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-9cgc8"]
Jul 17 01:08:48.962700 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:48.962664    2573 scope.go:117] "RemoveContainer" containerID="ac618e7b78963fc251d80531276db6075bd64c0752ce7beb080549cf3ba561da"
Jul 17 01:08:49.216127 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.216026    2573 generic.go:356] "Generic (PLEG): container finished" podID="51301041-e846-4cfc-8f1b-a876b52d4434" containerID="e98f3518d2747cf604c18e788b9f8cca531272ec0c01aaac84031b3e5811264f" exitCode=6
Jul 17 01:08:49.216653 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.216117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" event={"ID":"51301041-e846-4cfc-8f1b-a876b52d4434","Type":"ContainerDied","Data":"e98f3518d2747cf604c18e788b9f8cca531272ec0c01aaac84031b3e5811264f"}
Jul 17 01:08:49.216653 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.216188    2573 scope.go:117] "RemoveContainer" containerID="f836676ae3d0ac3cf73a32cd0bfac5b23987dfd62d05f17f528d877283a4e713"
Jul 17 01:08:49.216653 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.216475    2573 scope.go:117] "RemoveContainer" containerID="e98f3518d2747cf604c18e788b9f8cca531272ec0c01aaac84031b3e5811264f"
Jul 17 01:08:49.216852 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:49.216772    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-wfr86_odh-ai-gateway-infra(51301041-e846-4cfc-8f1b-a876b52d4434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" podUID="51301041-e846-4cfc-8f1b-a876b52d4434"
Jul 17 01:08:49.219559 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.219531    2573 generic.go:356] "Generic (PLEG): container finished" podID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerID="d572fc0ab1bd429b2144d93422b99d7d7337dbd0d86dd5d0fabe84180c3d2d1c" exitCode=6
Jul 17 01:08:49.219721 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.219581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" event={"ID":"3fe0e4d8-74e6-4030-92ad-c4673cd69def","Type":"ContainerDied","Data":"d572fc0ab1bd429b2144d93422b99d7d7337dbd0d86dd5d0fabe84180c3d2d1c"}
Jul 17 01:08:49.219988 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.219971    2573 scope.go:117] "RemoveContainer" containerID="d572fc0ab1bd429b2144d93422b99d7d7337dbd0d86dd5d0fabe84180c3d2d1c"
Jul 17 01:08:49.220191 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:49.220175    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq_odh-ai-gateway-infra(3fe0e4d8-74e6-4030-92ad-c4673cd69def)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def"
Jul 17 01:08:49.230199 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.229937    2573 scope.go:117] "RemoveContainer" containerID="b41847bf414efc294d4c5b98f0ec2960a176478fc2a8f4a3991f34a74e6dba12"
Jul 17 01:08:49.969589 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:49.969554    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" path="/var/lib/kubelet/pods/c1b143fb-4421-4d87-98e2-dc747076ec8d/volumes"
Jul 17 01:08:50.228037 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.228006    2573 scope.go:117] "RemoveContainer" containerID="e98f3518d2747cf604c18e788b9f8cca531272ec0c01aaac84031b3e5811264f"
Jul 17 01:08:50.228564 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:50.228280    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-wfr86_odh-ai-gateway-infra(51301041-e846-4cfc-8f1b-a876b52d4434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" podUID="51301041-e846-4cfc-8f1b-a876b52d4434"
Jul 17 01:08:50.230184 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.230153    2573 generic.go:356] "Generic (PLEG): container finished" podID="79288127-53ee-4047-91bf-a0488116bf84" containerID="5d1a057ca408f0ae85362e9da9b464f7c708d87a4a4414d0afa537739dfd50b9" exitCode=6
Jul 17 01:08:50.230476 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.230224    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" event={"ID":"79288127-53ee-4047-91bf-a0488116bf84","Type":"ContainerDied","Data":"5d1a057ca408f0ae85362e9da9b464f7c708d87a4a4414d0afa537739dfd50b9"}
Jul 17 01:08:50.230476 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.230285    2573 scope.go:117] "RemoveContainer" containerID="ac618e7b78963fc251d80531276db6075bd64c0752ce7beb080549cf3ba561da"
Jul 17 01:08:50.230666 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.230646    2573 scope.go:117] "RemoveContainer" containerID="5d1a057ca408f0ae85362e9da9b464f7c708d87a4a4414d0afa537739dfd50b9"
Jul 17 01:08:50.230929 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:08:50.230909    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r_odh-ai-gateway-infra(79288127-53ee-4047-91bf-a0488116bf84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" podUID="79288127-53ee-4047-91bf-a0488116bf84"
Jul 17 01:08:50.251406 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.251356    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"]
Jul 17 01:08:50.412458 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.412422    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"
Jul 17 01:08:50.592343 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.592295    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe0e4d8-74e6-4030-92ad-c4673cd69def-maas-api-service-ca\") pod \"3fe0e4d8-74e6-4030-92ad-c4673cd69def\" (UID: \"3fe0e4d8-74e6-4030-92ad-c4673cd69def\") "
Jul 17 01:08:50.592674 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.592650    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe0e4d8-74e6-4030-92ad-c4673cd69def-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fe0e4d8-74e6-4030-92ad-c4673cd69def" (UID: "3fe0e4d8-74e6-4030-92ad-c4673cd69def"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:08:50.693735 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:50.693679    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe0e4d8-74e6-4030-92ad-c4673cd69def-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:08:51.242603 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.242569    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"
Jul 17 01:08:51.242603 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.242585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq" event={"ID":"3fe0e4d8-74e6-4030-92ad-c4673cd69def","Type":"ContainerDied","Data":"1b33d180d45d01250205924bd3156429c2e859fd37025c40addc61003744241d"}
Jul 17 01:08:51.243217 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.242661    2573 scope.go:117] "RemoveContainer" containerID="d572fc0ab1bd429b2144d93422b99d7d7337dbd0d86dd5d0fabe84180c3d2d1c"
Jul 17 01:08:51.271401 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.271359    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"]
Jul 17 01:08:51.274779 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.274744    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"]
Jul 17 01:08:51.277734 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.277699    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-87rpq"]
Jul 17 01:08:51.437610 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.437564    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"
Jul 17 01:08:51.601614 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.601572    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79288127-53ee-4047-91bf-a0488116bf84-maas-api-service-ca\") pod \"79288127-53ee-4047-91bf-a0488116bf84\" (UID: \"79288127-53ee-4047-91bf-a0488116bf84\") "
Jul 17 01:08:51.602064 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.602035    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79288127-53ee-4047-91bf-a0488116bf84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79288127-53ee-4047-91bf-a0488116bf84" (UID: "79288127-53ee-4047-91bf-a0488116bf84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:08:51.702458 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.702411    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79288127-53ee-4047-91bf-a0488116bf84-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:08:51.968813 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:51.968692    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" path="/var/lib/kubelet/pods/3fe0e4d8-74e6-4030-92ad-c4673cd69def/volumes"
Jul 17 01:08:52.253488 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:52.253398    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"
Jul 17 01:08:52.253488 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:52.253400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r" event={"ID":"79288127-53ee-4047-91bf-a0488116bf84","Type":"ContainerDied","Data":"6f14c93063f4f76561a616ce7149d3161c39267a12ac548b046230dceb8bff3f"}
Jul 17 01:08:52.254113 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:52.253547    2573 scope.go:117] "RemoveContainer" containerID="5d1a057ca408f0ae85362e9da9b464f7c708d87a4a4414d0afa537739dfd50b9"
Jul 17 01:08:52.269247 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:52.269173    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"]
Jul 17 01:08:52.272549 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:52.272516    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-t7k4r"]
Jul 17 01:08:53.969006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:08:53.968963    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79288127-53ee-4047-91bf-a0488116bf84" path="/var/lib/kubelet/pods/79288127-53ee-4047-91bf-a0488116bf84/volumes"
Jul 17 01:09:05.966600 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:05.966561    2573 scope.go:117] "RemoveContainer" containerID="e98f3518d2747cf604c18e788b9f8cca531272ec0c01aaac84031b3e5811264f"
Jul 17 01:09:06.345502 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:06.345465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" event={"ID":"51301041-e846-4cfc-8f1b-a876b52d4434","Type":"ContainerStarted","Data":"b4cba709ee2c9ceb0355315a79887c01544ddb6963153431d04415b4c260337e"}
Jul 17 01:09:06.361781 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:06.361712    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" podStartSLOduration=20.36168945 podStartE2EDuration="20.36168945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:09:06.358573083 +0000 UTC m=+3140.934725659" watchObservedRunningTime="2026-07-17 01:09:06.36168945 +0000 UTC m=+3140.937842024"
Jul 17 01:09:07.353939 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:07.353886    2573 generic.go:356] "Generic (PLEG): container finished" podID="51301041-e846-4cfc-8f1b-a876b52d4434" containerID="b4cba709ee2c9ceb0355315a79887c01544ddb6963153431d04415b4c260337e" exitCode=6
Jul 17 01:09:07.354448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:07.353955    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" event={"ID":"51301041-e846-4cfc-8f1b-a876b52d4434","Type":"ContainerDied","Data":"b4cba709ee2c9ceb0355315a79887c01544ddb6963153431d04415b4c260337e"}
Jul 17 01:09:07.354448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:07.354024    2573 scope.go:117] "RemoveContainer" containerID="e98f3518d2747cf604c18e788b9f8cca531272ec0c01aaac84031b3e5811264f"
Jul 17 01:09:07.354448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:07.354326    2573 scope.go:117] "RemoveContainer" containerID="b4cba709ee2c9ceb0355315a79887c01544ddb6963153431d04415b4c260337e"
Jul 17 01:09:07.354646 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:07.354615    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-wfr86_odh-ai-gateway-infra(51301041-e846-4cfc-8f1b-a876b52d4434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" podUID="51301041-e846-4cfc-8f1b-a876b52d4434"
Jul 17 01:09:07.369960 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:07.369926    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"]
Jul 17 01:09:08.513493 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:08.513453    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"
Jul 17 01:09:08.569499 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:08.569453    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51301041-e846-4cfc-8f1b-a876b52d4434-maas-api-service-ca\") pod \"51301041-e846-4cfc-8f1b-a876b52d4434\" (UID: \"51301041-e846-4cfc-8f1b-a876b52d4434\") "
Jul 17 01:09:08.569973 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:08.569949    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51301041-e846-4cfc-8f1b-a876b52d4434-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51301041-e846-4cfc-8f1b-a876b52d4434" (UID: "51301041-e846-4cfc-8f1b-a876b52d4434"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:09:08.671209 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:08.671103    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51301041-e846-4cfc-8f1b-a876b52d4434-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:09:09.369397 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:09.369357    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"
Jul 17 01:09:09.369397 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:09.369382    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86" event={"ID":"51301041-e846-4cfc-8f1b-a876b52d4434","Type":"ContainerDied","Data":"e1678627e05f11f82d3a92e69449193fcca44d5423c4f593544e61fc8e7fc6ed"}
Jul 17 01:09:09.369705 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:09.369454    2573 scope.go:117] "RemoveContainer" containerID="b4cba709ee2c9ceb0355315a79887c01544ddb6963153431d04415b4c260337e"
Jul 17 01:09:09.394734 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:09.394679    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"]
Jul 17 01:09:09.398458 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:09.398420    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-wfr86"]
Jul 17 01:09:09.969550 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:09.969498    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" path="/var/lib/kubelet/pods/51301041-e846-4cfc-8f1b-a876b52d4434/volumes"
Jul 17 01:09:19.328865 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.328819    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"]
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329249    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329260    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329269    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329274    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329281    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329286    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329299    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329303    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329312    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329317    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329327    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329332    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329338    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329343    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:19.329341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329350    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329355    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329364    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329368    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329425    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329432    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329437    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329444    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329449    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329454    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329459    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329464    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:19.329869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.329470    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:19.334231 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.334193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"
Jul 17 01:09:19.336228 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.336201    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:09:19.338867 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.338829    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"]
Jul 17 01:09:19.476093 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.476043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c87c9a-e263-473c-a267-0be2dd35797a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-tl27t\" (UID: \"07c87c9a-e263-473c-a267-0be2dd35797a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"
Jul 17 01:09:19.576851 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.576782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c87c9a-e263-473c-a267-0be2dd35797a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-tl27t\" (UID: \"07c87c9a-e263-473c-a267-0be2dd35797a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"
Jul 17 01:09:19.577448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.577423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c87c9a-e263-473c-a267-0be2dd35797a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-tl27t\" (UID: \"07c87c9a-e263-473c-a267-0be2dd35797a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"
Jul 17 01:09:19.650467 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.650355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"
Jul 17 01:09:19.804634 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:19.804590    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"]
Jul 17 01:09:19.805374 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:09:19.805336    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c87c9a_e263_473c_a267_0be2dd35797a.slice/crio-3a7d6ea3247b8d7c484eaccd0f32178fd0eb13677745f271663d776dd77e2ef1 WatchSource:0}: Error finding container 3a7d6ea3247b8d7c484eaccd0f32178fd0eb13677745f271663d776dd77e2ef1: Status 404 returned error can't find the container with id 3a7d6ea3247b8d7c484eaccd0f32178fd0eb13677745f271663d776dd77e2ef1
Jul 17 01:09:20.441080 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:20.440968    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" event={"ID":"07c87c9a-e263-473c-a267-0be2dd35797a","Type":"ContainerStarted","Data":"d58ac683629f2fd4d603ce0cb87d4924c95e5a7d43d3fc2c308d8923b835332a"}
Jul 17 01:09:20.441080 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:20.441031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" event={"ID":"07c87c9a-e263-473c-a267-0be2dd35797a","Type":"ContainerStarted","Data":"3a7d6ea3247b8d7c484eaccd0f32178fd0eb13677745f271663d776dd77e2ef1"}
Jul 17 01:09:20.457199 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:20.457131    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" podStartSLOduration=1.457112135 podStartE2EDuration="1.457112135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:09:20.454738617 +0000 UTC m=+3155.030891192" watchObservedRunningTime="2026-07-17 01:09:20.457112135 +0000 UTC m=+3155.033264709"
Jul 17 01:09:21.448932 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:21.448887    2573 generic.go:356] "Generic (PLEG): container finished" podID="07c87c9a-e263-473c-a267-0be2dd35797a" containerID="d58ac683629f2fd4d603ce0cb87d4924c95e5a7d43d3fc2c308d8923b835332a" exitCode=6
Jul 17 01:09:21.449433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:21.448942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" event={"ID":"07c87c9a-e263-473c-a267-0be2dd35797a","Type":"ContainerDied","Data":"d58ac683629f2fd4d603ce0cb87d4924c95e5a7d43d3fc2c308d8923b835332a"}
Jul 17 01:09:21.451825 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:21.449915    2573 scope.go:117] "RemoveContainer" containerID="d58ac683629f2fd4d603ce0cb87d4924c95e5a7d43d3fc2c308d8923b835332a"
Jul 17 01:09:22.457810 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:22.457696    2573 generic.go:356] "Generic (PLEG): container finished" podID="07c87c9a-e263-473c-a267-0be2dd35797a" containerID="30d67a476fb88181cefb3cbab933b07af5e974b59cfc273b9c3ffa3b0d33b952" exitCode=6
Jul 17 01:09:22.457810 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:22.457779    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" event={"ID":"07c87c9a-e263-473c-a267-0be2dd35797a","Type":"ContainerDied","Data":"30d67a476fb88181cefb3cbab933b07af5e974b59cfc273b9c3ffa3b0d33b952"}
Jul 17 01:09:22.458347 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:22.457858    2573 scope.go:117] "RemoveContainer" containerID="d58ac683629f2fd4d603ce0cb87d4924c95e5a7d43d3fc2c308d8923b835332a"
Jul 17 01:09:22.458347 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:22.458052    2573 scope.go:117] "RemoveContainer" containerID="30d67a476fb88181cefb3cbab933b07af5e974b59cfc273b9c3ffa3b0d33b952"
Jul 17 01:09:22.458347 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:22.458293    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-tl27t_odh-ai-gateway-infra(07c87c9a-e263-473c-a267-0be2dd35797a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" podUID="07c87c9a-e263-473c-a267-0be2dd35797a"
Jul 17 01:09:23.467122 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.467090    2573 scope.go:117] "RemoveContainer" containerID="30d67a476fb88181cefb3cbab933b07af5e974b59cfc273b9c3ffa3b0d33b952"
Jul 17 01:09:23.467525 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:23.467385    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-tl27t_odh-ai-gateway-infra(07c87c9a-e263-473c-a267-0be2dd35797a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" podUID="07c87c9a-e263-473c-a267-0be2dd35797a"
Jul 17 01:09:23.607263 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607216    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"]
Jul 17 01:09:23.607694 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607665    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:23.607694 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607688    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:23.607934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607712    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:23.607934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607719    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe0e4d8-74e6-4030-92ad-c4673cd69def" containerName="revoke-keys"
Jul 17 01:09:23.607934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607732    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:23.607934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607738    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:23.607934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607825    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="79288127-53ee-4047-91bf-a0488116bf84" containerName="revoke-keys"
Jul 17 01:09:23.607934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607837    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="51301041-e846-4cfc-8f1b-a876b52d4434" containerName="revoke-keys"
Jul 17 01:09:23.607934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.607848    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b143fb-4421-4d87-98e2-dc747076ec8d" containerName="revoke-keys"
Jul 17 01:09:23.612308 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.612280    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"
Jul 17 01:09:23.616945 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.616908    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"]
Jul 17 01:09:23.720040 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.719925    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bfc8d1-2c8a-4bd5-a29d-18849c519e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4\" (UID: \"89bfc8d1-2c8a-4bd5-a29d-18849c519e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"
Jul 17 01:09:23.820666 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.820620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bfc8d1-2c8a-4bd5-a29d-18849c519e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4\" (UID: \"89bfc8d1-2c8a-4bd5-a29d-18849c519e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"
Jul 17 01:09:23.821402 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.821381    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bfc8d1-2c8a-4bd5-a29d-18849c519e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4\" (UID: \"89bfc8d1-2c8a-4bd5-a29d-18849c519e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"
Jul 17 01:09:23.927369 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:23.927319    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"
Jul 17 01:09:24.083560 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:24.083508    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"]
Jul 17 01:09:24.085299 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:09:24.085266    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bfc8d1_2c8a_4bd5_a29d_18849c519e98.slice/crio-56d0ff8e279f544367ae2c48a1b46df6fd30350abac72d90778693389eb090ec WatchSource:0}: Error finding container 56d0ff8e279f544367ae2c48a1b46df6fd30350abac72d90778693389eb090ec: Status 404 returned error can't find the container with id 56d0ff8e279f544367ae2c48a1b46df6fd30350abac72d90778693389eb090ec
Jul 17 01:09:24.475130 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:24.475088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" event={"ID":"89bfc8d1-2c8a-4bd5-a29d-18849c519e98","Type":"ContainerStarted","Data":"22ca09d70b06ac56640c484ac6f6f6b9e7b81ef5a66e697f4669ac92b024ede5"}
Jul 17 01:09:24.475563 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:24.475179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" event={"ID":"89bfc8d1-2c8a-4bd5-a29d-18849c519e98","Type":"ContainerStarted","Data":"56d0ff8e279f544367ae2c48a1b46df6fd30350abac72d90778693389eb090ec"}
Jul 17 01:09:24.491268 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:24.491179    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" podStartSLOduration=1.4911597890000001 podStartE2EDuration="1.491159789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:09:24.490844946 +0000 UTC m=+3159.066997522" watchObservedRunningTime="2026-07-17 01:09:24.491159789 +0000 UTC m=+3159.067312363"
Jul 17 01:09:25.484062 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:25.484014    2573 generic.go:356] "Generic (PLEG): container finished" podID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerID="22ca09d70b06ac56640c484ac6f6f6b9e7b81ef5a66e697f4669ac92b024ede5" exitCode=6
Jul 17 01:09:25.484533 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:25.484061    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" event={"ID":"89bfc8d1-2c8a-4bd5-a29d-18849c519e98","Type":"ContainerDied","Data":"22ca09d70b06ac56640c484ac6f6f6b9e7b81ef5a66e697f4669ac92b024ede5"}
Jul 17 01:09:25.484533 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:25.484358    2573 scope.go:117] "RemoveContainer" containerID="22ca09d70b06ac56640c484ac6f6f6b9e7b81ef5a66e697f4669ac92b024ede5"
Jul 17 01:09:26.493257 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:26.493150    2573 generic.go:356] "Generic (PLEG): container finished" podID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerID="b91dceba5363c29c5e11246078e45f3b110b83165ab885742d32d1a0fdf0f3d8" exitCode=6
Jul 17 01:09:26.493700 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:26.493246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" event={"ID":"89bfc8d1-2c8a-4bd5-a29d-18849c519e98","Type":"ContainerDied","Data":"b91dceba5363c29c5e11246078e45f3b110b83165ab885742d32d1a0fdf0f3d8"}
Jul 17 01:09:26.493700 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:26.493315    2573 scope.go:117] "RemoveContainer" containerID="22ca09d70b06ac56640c484ac6f6f6b9e7b81ef5a66e697f4669ac92b024ede5"
Jul 17 01:09:26.493700 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:26.493583    2573 scope.go:117] "RemoveContainer" containerID="b91dceba5363c29c5e11246078e45f3b110b83165ab885742d32d1a0fdf0f3d8"
Jul 17 01:09:26.493904 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:26.493869    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4_odh-ai-gateway-infra(89bfc8d1-2c8a-4bd5-a29d-18849c519e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98"
Jul 17 01:09:27.501717 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:27.501685    2573 scope.go:117] "RemoveContainer" containerID="b91dceba5363c29c5e11246078e45f3b110b83165ab885742d32d1a0fdf0f3d8"
Jul 17 01:09:27.502188 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:27.501977    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4_odh-ai-gateway-infra(89bfc8d1-2c8a-4bd5-a29d-18849c519e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98"
Jul 17 01:09:31.180528 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.180483    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"]
Jul 17 01:09:31.185348 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.185321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"
Jul 17 01:09:31.190625 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.190584    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"]
Jul 17 01:09:31.295274 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.295224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3607883-8952-4dce-b9ee-2ba913b5a5b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s\" (UID: \"b3607883-8952-4dce-b9ee-2ba913b5a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"
Jul 17 01:09:31.396238 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.396188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3607883-8952-4dce-b9ee-2ba913b5a5b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s\" (UID: \"b3607883-8952-4dce-b9ee-2ba913b5a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"
Jul 17 01:09:31.396928 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.396907    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3607883-8952-4dce-b9ee-2ba913b5a5b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s\" (UID: \"b3607883-8952-4dce-b9ee-2ba913b5a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"
Jul 17 01:09:31.501074 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.500967    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"
Jul 17 01:09:31.656727 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:31.656695    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"]
Jul 17 01:09:31.659224 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:09:31.659181    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3607883_8952_4dce_b9ee_2ba913b5a5b8.slice/crio-32927907e0b3fe00a0e193e6437867a1d83994da2604f075833e25603f0d67f8 WatchSource:0}: Error finding container 32927907e0b3fe00a0e193e6437867a1d83994da2604f075833e25603f0d67f8: Status 404 returned error can't find the container with id 32927907e0b3fe00a0e193e6437867a1d83994da2604f075833e25603f0d67f8
Jul 17 01:09:32.535661 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:32.535620    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerID="8b99a87997079132b7f644838ecbeb6db1f2877bc37a757207fd444431b7db62" exitCode=6
Jul 17 01:09:32.536126 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:32.535705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" event={"ID":"b3607883-8952-4dce-b9ee-2ba913b5a5b8","Type":"ContainerDied","Data":"8b99a87997079132b7f644838ecbeb6db1f2877bc37a757207fd444431b7db62"}
Jul 17 01:09:32.536126 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:32.535760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" event={"ID":"b3607883-8952-4dce-b9ee-2ba913b5a5b8","Type":"ContainerStarted","Data":"32927907e0b3fe00a0e193e6437867a1d83994da2604f075833e25603f0d67f8"}
Jul 17 01:09:32.536126 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:32.535944    2573 scope.go:117] "RemoveContainer" containerID="8b99a87997079132b7f644838ecbeb6db1f2877bc37a757207fd444431b7db62"
Jul 17 01:09:33.546273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:33.546239    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerID="0cb7cda14570ce96e4e6bfe84a741d3ebd8dd57a96517e64c6eed85d010f1353" exitCode=6
Jul 17 01:09:33.546754 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:33.546326    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" event={"ID":"b3607883-8952-4dce-b9ee-2ba913b5a5b8","Type":"ContainerDied","Data":"0cb7cda14570ce96e4e6bfe84a741d3ebd8dd57a96517e64c6eed85d010f1353"}
Jul 17 01:09:33.546754 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:33.546401    2573 scope.go:117] "RemoveContainer" containerID="8b99a87997079132b7f644838ecbeb6db1f2877bc37a757207fd444431b7db62"
Jul 17 01:09:33.546754 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:33.546666    2573 scope.go:117] "RemoveContainer" containerID="0cb7cda14570ce96e4e6bfe84a741d3ebd8dd57a96517e64c6eed85d010f1353"
Jul 17 01:09:33.546978 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:33.546940    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s_odh-ai-gateway-infra(b3607883-8952-4dce-b9ee-2ba913b5a5b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8"
Jul 17 01:09:34.556097 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:34.556053    2573 scope.go:117] "RemoveContainer" containerID="0cb7cda14570ce96e4e6bfe84a741d3ebd8dd57a96517e64c6eed85d010f1353"
Jul 17 01:09:34.556601 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:34.556356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s_odh-ai-gateway-infra(b3607883-8952-4dce-b9ee-2ba913b5a5b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8"
Jul 17 01:09:38.962551 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:38.962517    2573 scope.go:117] "RemoveContainer" containerID="30d67a476fb88181cefb3cbab933b07af5e974b59cfc273b9c3ffa3b0d33b952"
Jul 17 01:09:39.589784 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:39.589748    2573 generic.go:356] "Generic (PLEG): container finished" podID="07c87c9a-e263-473c-a267-0be2dd35797a" containerID="8d29d8dfa54b17a43995a08d5ae588e73dfff985bc4ccd79051333b3bdc70589" exitCode=6
Jul 17 01:09:39.590009 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:39.589824    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" event={"ID":"07c87c9a-e263-473c-a267-0be2dd35797a","Type":"ContainerDied","Data":"8d29d8dfa54b17a43995a08d5ae588e73dfff985bc4ccd79051333b3bdc70589"}
Jul 17 01:09:39.590009 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:39.589904    2573 scope.go:117] "RemoveContainer" containerID="30d67a476fb88181cefb3cbab933b07af5e974b59cfc273b9c3ffa3b0d33b952"
Jul 17 01:09:39.590276 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:39.590245    2573 scope.go:117] "RemoveContainer" containerID="8d29d8dfa54b17a43995a08d5ae588e73dfff985bc4ccd79051333b3bdc70589"
Jul 17 01:09:39.590538 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:39.590514    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-tl27t_odh-ai-gateway-infra(07c87c9a-e263-473c-a267-0be2dd35797a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" podUID="07c87c9a-e263-473c-a267-0be2dd35797a"
Jul 17 01:09:40.615768 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:40.615719    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"]
Jul 17 01:09:40.767454 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:40.767409    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"
Jul 17 01:09:40.782785 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:40.782745    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c87c9a-e263-473c-a267-0be2dd35797a-maas-api-service-ca\") pod \"07c87c9a-e263-473c-a267-0be2dd35797a\" (UID: \"07c87c9a-e263-473c-a267-0be2dd35797a\") "
Jul 17 01:09:40.783553 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:40.783521    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c87c9a-e263-473c-a267-0be2dd35797a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07c87c9a-e263-473c-a267-0be2dd35797a" (UID: "07c87c9a-e263-473c-a267-0be2dd35797a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:09:40.883971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:40.883868    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c87c9a-e263-473c-a267-0be2dd35797a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:09:41.607041 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:41.607005    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"
Jul 17 01:09:41.607268 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:41.607034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t" event={"ID":"07c87c9a-e263-473c-a267-0be2dd35797a","Type":"ContainerDied","Data":"3a7d6ea3247b8d7c484eaccd0f32178fd0eb13677745f271663d776dd77e2ef1"}
Jul 17 01:09:41.607268 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:41.607141    2573 scope.go:117] "RemoveContainer" containerID="8d29d8dfa54b17a43995a08d5ae588e73dfff985bc4ccd79051333b3bdc70589"
Jul 17 01:09:41.632673 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:41.632630    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"]
Jul 17 01:09:41.638929 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:41.638884    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-tl27t"]
Jul 17 01:09:41.962479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:41.962389    2573 scope.go:117] "RemoveContainer" containerID="b91dceba5363c29c5e11246078e45f3b110b83165ab885742d32d1a0fdf0f3d8"
Jul 17 01:09:41.969310 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:41.969281    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" path="/var/lib/kubelet/pods/07c87c9a-e263-473c-a267-0be2dd35797a/volumes"
Jul 17 01:09:42.617463 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:42.617423    2573 generic.go:356] "Generic (PLEG): container finished" podID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerID="23fc034cf666c8cab93469356de8ab309216deb0faade2374227a30c72aa9d29" exitCode=6
Jul 17 01:09:42.617673 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:42.617493    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" event={"ID":"89bfc8d1-2c8a-4bd5-a29d-18849c519e98","Type":"ContainerDied","Data":"23fc034cf666c8cab93469356de8ab309216deb0faade2374227a30c72aa9d29"}
Jul 17 01:09:42.617673 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:42.617559    2573 scope.go:117] "RemoveContainer" containerID="b91dceba5363c29c5e11246078e45f3b110b83165ab885742d32d1a0fdf0f3d8"
Jul 17 01:09:42.617958 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:42.617938    2573 scope.go:117] "RemoveContainer" containerID="23fc034cf666c8cab93469356de8ab309216deb0faade2374227a30c72aa9d29"
Jul 17 01:09:42.618218 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:42.618191    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4_odh-ai-gateway-infra(89bfc8d1-2c8a-4bd5-a29d-18849c519e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98"
Jul 17 01:09:43.645277 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:43.645225    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"]
Jul 17 01:09:43.801434 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:43.801399    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"
Jul 17 01:09:43.810933 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:43.810891    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bfc8d1-2c8a-4bd5-a29d-18849c519e98-maas-api-service-ca\") pod \"89bfc8d1-2c8a-4bd5-a29d-18849c519e98\" (UID: \"89bfc8d1-2c8a-4bd5-a29d-18849c519e98\") "
Jul 17 01:09:43.811315 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:43.811288    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bfc8d1-2c8a-4bd5-a29d-18849c519e98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89bfc8d1-2c8a-4bd5-a29d-18849c519e98" (UID: "89bfc8d1-2c8a-4bd5-a29d-18849c519e98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:09:43.911632 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:43.911532    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bfc8d1-2c8a-4bd5-a29d-18849c519e98-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:09:44.634736 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:44.634703    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"
Jul 17 01:09:44.635012 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:44.634692    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4" event={"ID":"89bfc8d1-2c8a-4bd5-a29d-18849c519e98","Type":"ContainerDied","Data":"56d0ff8e279f544367ae2c48a1b46df6fd30350abac72d90778693389eb090ec"}
Jul 17 01:09:44.635012 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:44.634865    2573 scope.go:117] "RemoveContainer" containerID="23fc034cf666c8cab93469356de8ab309216deb0faade2374227a30c72aa9d29"
Jul 17 01:09:44.650460 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:44.650417    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"]
Jul 17 01:09:44.655418 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:44.655375    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-lj6x4"]
Jul 17 01:09:45.970539 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:45.970500    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" path="/var/lib/kubelet/pods/89bfc8d1-2c8a-4bd5-a29d-18849c519e98/volumes"
Jul 17 01:09:46.646836 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.646773    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"]
Jul 17 01:09:46.647242 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647219    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.647242 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647241    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647251    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647259    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647273    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647281    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647290    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647298    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647322    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647329    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647340    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647349    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647422    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.647423 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647429    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.648059 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647435    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:09:46.648059 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647444    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.648059 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.647450    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="89bfc8d1-2c8a-4bd5-a29d-18849c519e98" containerName="revoke-keys"
Jul 17 01:09:46.652142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.652111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"
Jul 17 01:09:46.657533 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.657494    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"]
Jul 17 01:09:46.738838 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.738764    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1011691-3989-4f4b-84d1-f65a81b4b714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-tgmlf\" (UID: \"b1011691-3989-4f4b-84d1-f65a81b4b714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"
Jul 17 01:09:46.839444 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.839376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1011691-3989-4f4b-84d1-f65a81b4b714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-tgmlf\" (UID: \"b1011691-3989-4f4b-84d1-f65a81b4b714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"
Jul 17 01:09:46.840136 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.840113    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1011691-3989-4f4b-84d1-f65a81b4b714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-tgmlf\" (UID: \"b1011691-3989-4f4b-84d1-f65a81b4b714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"
Jul 17 01:09:46.962398 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.962287    2573 scope.go:117] "RemoveContainer" containerID="0cb7cda14570ce96e4e6bfe84a741d3ebd8dd57a96517e64c6eed85d010f1353"
Jul 17 01:09:46.967628 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:46.967596    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"
Jul 17 01:09:47.136362 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:47.136319    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"]
Jul 17 01:09:47.138186 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:09:47.138144    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1011691_3989_4f4b_84d1_f65a81b4b714.slice/crio-921a782f04f6f12c3587d412001e0aeb80a87901d78ea5a94b203b98e01878ec WatchSource:0}: Error finding container 921a782f04f6f12c3587d412001e0aeb80a87901d78ea5a94b203b98e01878ec: Status 404 returned error can't find the container with id 921a782f04f6f12c3587d412001e0aeb80a87901d78ea5a94b203b98e01878ec
Jul 17 01:09:47.663391 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:47.663285    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" event={"ID":"b3607883-8952-4dce-b9ee-2ba913b5a5b8","Type":"ContainerStarted","Data":"d22d19ea5e7d882561bd126c16f13a7c69f6c96b6ba6fbdc91abc006425ca926"}
Jul 17 01:09:47.665001 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:47.664967    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" event={"ID":"b1011691-3989-4f4b-84d1-f65a81b4b714","Type":"ContainerStarted","Data":"4639ce0222a61b8e007619cf15f82341894004fdba5b0c97a48aa7f21cb35d05"}
Jul 17 01:09:47.665187 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:47.665028    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" event={"ID":"b1011691-3989-4f4b-84d1-f65a81b4b714","Type":"ContainerStarted","Data":"921a782f04f6f12c3587d412001e0aeb80a87901d78ea5a94b203b98e01878ec"}
Jul 17 01:09:47.679940 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:47.679819    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" podStartSLOduration=16.67977392 podStartE2EDuration="16.67977392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:09:47.676094296 +0000 UTC m=+3182.252246899" watchObservedRunningTime="2026-07-17 01:09:47.67977392 +0000 UTC m=+3182.255926498"
Jul 17 01:09:47.695450 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:47.695387    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" podStartSLOduration=1.695367804 podStartE2EDuration="1.695367804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:09:47.692995535 +0000 UTC m=+3182.269148110" watchObservedRunningTime="2026-07-17 01:09:47.695367804 +0000 UTC m=+3182.271520429"
Jul 17 01:09:48.673522 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.673481    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerID="d22d19ea5e7d882561bd126c16f13a7c69f6c96b6ba6fbdc91abc006425ca926" exitCode=6
Jul 17 01:09:48.674057 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.673554    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" event={"ID":"b3607883-8952-4dce-b9ee-2ba913b5a5b8","Type":"ContainerDied","Data":"d22d19ea5e7d882561bd126c16f13a7c69f6c96b6ba6fbdc91abc006425ca926"}
Jul 17 01:09:48.674057 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.673635    2573 scope.go:117] "RemoveContainer" containerID="0cb7cda14570ce96e4e6bfe84a741d3ebd8dd57a96517e64c6eed85d010f1353"
Jul 17 01:09:48.674057 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.673947    2573 scope.go:117] "RemoveContainer" containerID="d22d19ea5e7d882561bd126c16f13a7c69f6c96b6ba6fbdc91abc006425ca926"
Jul 17 01:09:48.674272 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:48.674252    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s_odh-ai-gateway-infra(b3607883-8952-4dce-b9ee-2ba913b5a5b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8"
Jul 17 01:09:48.675384 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.675362    2573 generic.go:356] "Generic (PLEG): container finished" podID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerID="4639ce0222a61b8e007619cf15f82341894004fdba5b0c97a48aa7f21cb35d05" exitCode=6
Jul 17 01:09:48.675510 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.675442    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" event={"ID":"b1011691-3989-4f4b-84d1-f65a81b4b714","Type":"ContainerDied","Data":"4639ce0222a61b8e007619cf15f82341894004fdba5b0c97a48aa7f21cb35d05"}
Jul 17 01:09:48.675691 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.675678    2573 scope.go:117] "RemoveContainer" containerID="4639ce0222a61b8e007619cf15f82341894004fdba5b0c97a48aa7f21cb35d05"
Jul 17 01:09:48.688928 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:48.688884    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"]
Jul 17 01:09:49.686526 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.686477    2573 generic.go:356] "Generic (PLEG): container finished" podID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerID="08db4b9ad3fe073899f8939517298ba136b5e1b21ef3eebcddf8276ce03598d3" exitCode=6
Jul 17 01:09:49.687022 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.686624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" event={"ID":"b1011691-3989-4f4b-84d1-f65a81b4b714","Type":"ContainerDied","Data":"08db4b9ad3fe073899f8939517298ba136b5e1b21ef3eebcddf8276ce03598d3"}
Jul 17 01:09:49.687022 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.686714    2573 scope.go:117] "RemoveContainer" containerID="4639ce0222a61b8e007619cf15f82341894004fdba5b0c97a48aa7f21cb35d05"
Jul 17 01:09:49.687022 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.687007    2573 scope.go:117] "RemoveContainer" containerID="08db4b9ad3fe073899f8939517298ba136b5e1b21ef3eebcddf8276ce03598d3"
Jul 17 01:09:49.687374 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:49.687334    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-tgmlf_odh-ai-gateway-infra(b1011691-3989-4f4b-84d1-f65a81b4b714)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714"
Jul 17 01:09:49.835645 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.835615    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"
Jul 17 01:09:49.870097 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.870060    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3607883-8952-4dce-b9ee-2ba913b5a5b8-maas-api-service-ca\") pod \"b3607883-8952-4dce-b9ee-2ba913b5a5b8\" (UID: \"b3607883-8952-4dce-b9ee-2ba913b5a5b8\") "
Jul 17 01:09:49.870532 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.870501    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3607883-8952-4dce-b9ee-2ba913b5a5b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3607883-8952-4dce-b9ee-2ba913b5a5b8" (UID: "b3607883-8952-4dce-b9ee-2ba913b5a5b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:09:49.971070 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:49.970968    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3607883-8952-4dce-b9ee-2ba913b5a5b8-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:09:50.694776 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:50.694732    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s" event={"ID":"b3607883-8952-4dce-b9ee-2ba913b5a5b8","Type":"ContainerDied","Data":"32927907e0b3fe00a0e193e6437867a1d83994da2604f075833e25603f0d67f8"}
Jul 17 01:09:50.695368 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:50.694742    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"
Jul 17 01:09:50.695368 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:50.694846    2573 scope.go:117] "RemoveContainer" containerID="d22d19ea5e7d882561bd126c16f13a7c69f6c96b6ba6fbdc91abc006425ca926"
Jul 17 01:09:50.697144 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:50.697106    2573 scope.go:117] "RemoveContainer" containerID="08db4b9ad3fe073899f8939517298ba136b5e1b21ef3eebcddf8276ce03598d3"
Jul 17 01:09:50.697420 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:09:50.697391    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-tgmlf_odh-ai-gateway-infra(b1011691-3989-4f4b-84d1-f65a81b4b714)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714"
Jul 17 01:09:50.709892 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:50.709834    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"]
Jul 17 01:09:50.715981 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:50.715942    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-j6g7s"]
Jul 17 01:09:51.969223 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:09:51.969185    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" path="/var/lib/kubelet/pods/b3607883-8952-4dce-b9ee-2ba913b5a5b8/volumes"
Jul 17 01:10:02.963213 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:02.963172    2573 scope.go:117] "RemoveContainer" containerID="08db4b9ad3fe073899f8939517298ba136b5e1b21ef3eebcddf8276ce03598d3"
Jul 17 01:10:03.784874 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:03.784835    2573 generic.go:356] "Generic (PLEG): container finished" podID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerID="a6514a3aa5115201f9213a8ed8b589e8875b81775169899fb9d2fc143db09612" exitCode=6
Jul 17 01:10:03.785079 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:03.784914    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" event={"ID":"b1011691-3989-4f4b-84d1-f65a81b4b714","Type":"ContainerDied","Data":"a6514a3aa5115201f9213a8ed8b589e8875b81775169899fb9d2fc143db09612"}
Jul 17 01:10:03.785079 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:03.784982    2573 scope.go:117] "RemoveContainer" containerID="08db4b9ad3fe073899f8939517298ba136b5e1b21ef3eebcddf8276ce03598d3"
Jul 17 01:10:03.785394 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:03.785371    2573 scope.go:117] "RemoveContainer" containerID="a6514a3aa5115201f9213a8ed8b589e8875b81775169899fb9d2fc143db09612"
Jul 17 01:10:03.785655 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:03.785635    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-tgmlf_odh-ai-gateway-infra(b1011691-3989-4f4b-84d1-f65a81b4b714)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714"
Jul 17 01:10:04.809652 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:04.809600    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"]
Jul 17 01:10:04.970306 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:04.970268    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"
Jul 17 01:10:05.120104 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.119994    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1011691-3989-4f4b-84d1-f65a81b4b714-maas-api-service-ca\") pod \"b1011691-3989-4f4b-84d1-f65a81b4b714\" (UID: \"b1011691-3989-4f4b-84d1-f65a81b4b714\") "
Jul 17 01:10:05.120404 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.120378    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1011691-3989-4f4b-84d1-f65a81b4b714-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1011691-3989-4f4b-84d1-f65a81b4b714" (UID: "b1011691-3989-4f4b-84d1-f65a81b4b714"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:10:05.221681 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.221633    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1011691-3989-4f4b-84d1-f65a81b4b714-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:10:05.800624 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.800590    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"
Jul 17 01:10:05.800852 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.800584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf" event={"ID":"b1011691-3989-4f4b-84d1-f65a81b4b714","Type":"ContainerDied","Data":"921a782f04f6f12c3587d412001e0aeb80a87901d78ea5a94b203b98e01878ec"}
Jul 17 01:10:05.800852 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.800739    2573 scope.go:117] "RemoveContainer" containerID="a6514a3aa5115201f9213a8ed8b589e8875b81775169899fb9d2fc143db09612"
Jul 17 01:10:05.831143 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.831092    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"]
Jul 17 01:10:05.833982 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.833946    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-tgmlf"]
Jul 17 01:10:05.971260 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:05.971221    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" path="/var/lib/kubelet/pods/b1011691-3989-4f4b-84d1-f65a81b4b714/volumes"
Jul 17 01:10:19.373600 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.373543    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"]
Jul 17 01:10:19.374254 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374188    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:19.374254 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374214    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:19.374254 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374225    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374254 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374235    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374259    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374268    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374282    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374291    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374303    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374311    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374405    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374416    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374428    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374440    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1011691-3989-4f4b-84d1-f65a81b4b714" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374449    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:19.374479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.374460    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c87c9a-e263-473c-a267-0be2dd35797a" containerName="revoke-keys"
Jul 17 01:10:19.378005 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.377977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"
Jul 17 01:10:19.380047 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.380015    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:10:19.383397 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.383355    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"]
Jul 17 01:10:19.565322 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.565274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44d5c-1146-49ec-8720-f3250b3c00db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4\" (UID: \"83c44d5c-1146-49ec-8720-f3250b3c00db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"
Jul 17 01:10:19.666338 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.666234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44d5c-1146-49ec-8720-f3250b3c00db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4\" (UID: \"83c44d5c-1146-49ec-8720-f3250b3c00db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"
Jul 17 01:10:19.667040 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.667015    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44d5c-1146-49ec-8720-f3250b3c00db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4\" (UID: \"83c44d5c-1146-49ec-8720-f3250b3c00db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"
Jul 17 01:10:19.692269 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.692224    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"
Jul 17 01:10:19.847337 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.847305    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"]
Jul 17 01:10:19.849096 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:10:19.849055    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c44d5c_1146_49ec_8720_f3250b3c00db.slice/crio-06315edb13d0feee553e51acc024723eb11859b6ea5be17b1c39124360c3f203 WatchSource:0}: Error finding container 06315edb13d0feee553e51acc024723eb11859b6ea5be17b1c39124360c3f203: Status 404 returned error can't find the container with id 06315edb13d0feee553e51acc024723eb11859b6ea5be17b1c39124360c3f203
Jul 17 01:10:19.891664 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:19.891621    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" event={"ID":"83c44d5c-1146-49ec-8720-f3250b3c00db","Type":"ContainerStarted","Data":"06315edb13d0feee553e51acc024723eb11859b6ea5be17b1c39124360c3f203"}
Jul 17 01:10:20.899359 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:20.899319    2573 generic.go:356] "Generic (PLEG): container finished" podID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerID="ff5553f0ae084195e9838a2f1e55d61c9479b0e8b94bf5d3e37080317f069d45" exitCode=6
Jul 17 01:10:20.899821 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:20.899403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" event={"ID":"83c44d5c-1146-49ec-8720-f3250b3c00db","Type":"ContainerDied","Data":"ff5553f0ae084195e9838a2f1e55d61c9479b0e8b94bf5d3e37080317f069d45"}
Jul 17 01:10:20.899821 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:20.899582    2573 scope.go:117] "RemoveContainer" containerID="ff5553f0ae084195e9838a2f1e55d61c9479b0e8b94bf5d3e37080317f069d45"
Jul 17 01:10:21.224400 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.224352    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"]
Jul 17 01:10:21.224824 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.224785    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:21.224824 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.224817    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:21.224922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.224896    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3607883-8952-4dce-b9ee-2ba913b5a5b8" containerName="revoke-keys"
Jul 17 01:10:21.229387 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.229358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"
Jul 17 01:10:21.234767 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.234730    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"]
Jul 17 01:10:21.280283 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.280243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1859cc-0c24-4d25-a952-6dbdb54624bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl\" (UID: \"1a1859cc-0c24-4d25-a952-6dbdb54624bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"
Jul 17 01:10:21.381633 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.381583    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1859cc-0c24-4d25-a952-6dbdb54624bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl\" (UID: \"1a1859cc-0c24-4d25-a952-6dbdb54624bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"
Jul 17 01:10:21.382330 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.382306    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1859cc-0c24-4d25-a952-6dbdb54624bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl\" (UID: \"1a1859cc-0c24-4d25-a952-6dbdb54624bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"
Jul 17 01:10:21.545178 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.545139    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"
Jul 17 01:10:21.701502 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.701469    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"]
Jul 17 01:10:21.703332 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:10:21.703290    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1859cc_0c24_4d25_a952_6dbdb54624bd.slice/crio-8f56cebda54a94fa4e29b73822add5e05280d146f82c5997dba377a169abbe63 WatchSource:0}: Error finding container 8f56cebda54a94fa4e29b73822add5e05280d146f82c5997dba377a169abbe63: Status 404 returned error can't find the container with id 8f56cebda54a94fa4e29b73822add5e05280d146f82c5997dba377a169abbe63
Jul 17 01:10:21.907296 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.907193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" event={"ID":"1a1859cc-0c24-4d25-a952-6dbdb54624bd","Type":"ContainerStarted","Data":"8f56cebda54a94fa4e29b73822add5e05280d146f82c5997dba377a169abbe63"}
Jul 17 01:10:21.909304 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.909267    2573 generic.go:356] "Generic (PLEG): container finished" podID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerID="2406b86af96a8dc21622fd11792676006a1cf8621a4d0502921bbf11e1bccb86" exitCode=6
Jul 17 01:10:21.909511 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.909339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" event={"ID":"83c44d5c-1146-49ec-8720-f3250b3c00db","Type":"ContainerDied","Data":"2406b86af96a8dc21622fd11792676006a1cf8621a4d0502921bbf11e1bccb86"}
Jul 17 01:10:21.909511 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.909398    2573 scope.go:117] "RemoveContainer" containerID="ff5553f0ae084195e9838a2f1e55d61c9479b0e8b94bf5d3e37080317f069d45"
Jul 17 01:10:21.909657 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:21.909630    2573 scope.go:117] "RemoveContainer" containerID="2406b86af96a8dc21622fd11792676006a1cf8621a4d0502921bbf11e1bccb86"
Jul 17 01:10:21.910004 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:21.909981    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4_odh-ai-gateway-infra(83c44d5c-1146-49ec-8720-f3250b3c00db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db"
Jul 17 01:10:22.917958 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:22.917924    2573 scope.go:117] "RemoveContainer" containerID="2406b86af96a8dc21622fd11792676006a1cf8621a4d0502921bbf11e1bccb86"
Jul 17 01:10:22.918433 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:22.918162    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4_odh-ai-gateway-infra(83c44d5c-1146-49ec-8720-f3250b3c00db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db"
Jul 17 01:10:22.919553 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:22.919517    2573 generic.go:356] "Generic (PLEG): container finished" podID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerID="8566bcd3d07fc681b829aad9413ce169461a703773fd8f58f241bccf7a38120f" exitCode=6
Jul 17 01:10:22.919692 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:22.919593    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" event={"ID":"1a1859cc-0c24-4d25-a952-6dbdb54624bd","Type":"ContainerDied","Data":"8566bcd3d07fc681b829aad9413ce169461a703773fd8f58f241bccf7a38120f"}
Jul 17 01:10:22.919841 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:22.919826    2573 scope.go:117] "RemoveContainer" containerID="8566bcd3d07fc681b829aad9413ce169461a703773fd8f58f241bccf7a38120f"
Jul 17 01:10:23.652339 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.652292    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"]
Jul 17 01:10:23.656189 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.656161    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"
Jul 17 01:10:23.662090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.661987    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"]
Jul 17 01:10:23.698679 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.698628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f55227a-62b8-4163-a235-631cc4790424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4\" (UID: \"9f55227a-62b8-4163-a235-631cc4790424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"
Jul 17 01:10:23.799838 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.799768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f55227a-62b8-4163-a235-631cc4790424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4\" (UID: \"9f55227a-62b8-4163-a235-631cc4790424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"
Jul 17 01:10:23.800511 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.800486    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f55227a-62b8-4163-a235-631cc4790424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4\" (UID: \"9f55227a-62b8-4163-a235-631cc4790424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"
Jul 17 01:10:23.929017 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.928888    2573 generic.go:356] "Generic (PLEG): container finished" podID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerID="a6885f91fea363c82f539c9c17e5eb931f0e73772720f1fd296e003856d5851e" exitCode=6
Jul 17 01:10:23.929017 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.928986    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" event={"ID":"1a1859cc-0c24-4d25-a952-6dbdb54624bd","Type":"ContainerDied","Data":"a6885f91fea363c82f539c9c17e5eb931f0e73772720f1fd296e003856d5851e"}
Jul 17 01:10:23.929558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.929079    2573 scope.go:117] "RemoveContainer" containerID="8566bcd3d07fc681b829aad9413ce169461a703773fd8f58f241bccf7a38120f"
Jul 17 01:10:23.929558 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.929203    2573 scope.go:117] "RemoveContainer" containerID="a6885f91fea363c82f539c9c17e5eb931f0e73772720f1fd296e003856d5851e"
Jul 17 01:10:23.929558 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:23.929445    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl_odh-ai-gateway-infra(1a1859cc-0c24-4d25-a952-6dbdb54624bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd"
Jul 17 01:10:23.971369 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:23.971317    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"
Jul 17 01:10:24.131741 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:24.131547    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"]
Jul 17 01:10:24.134504 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:10:24.134463    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f55227a_62b8_4163_a235_631cc4790424.slice/crio-e5da82f028be1cd5547c6b729f8c6ebee675bff495087ca6e9ded5578c0309f6 WatchSource:0}: Error finding container e5da82f028be1cd5547c6b729f8c6ebee675bff495087ca6e9ded5578c0309f6: Status 404 returned error can't find the container with id e5da82f028be1cd5547c6b729f8c6ebee675bff495087ca6e9ded5578c0309f6
Jul 17 01:10:24.941011 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:24.940963    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f55227a-62b8-4163-a235-631cc4790424" containerID="edd8fa43e4302580f3fa6f8359bf3f0e2c73605083bb884810b5f4930802c568" exitCode=6
Jul 17 01:10:24.941498 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:24.941051    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" event={"ID":"9f55227a-62b8-4163-a235-631cc4790424","Type":"ContainerDied","Data":"edd8fa43e4302580f3fa6f8359bf3f0e2c73605083bb884810b5f4930802c568"}
Jul 17 01:10:24.941498 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:24.941160    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" event={"ID":"9f55227a-62b8-4163-a235-631cc4790424","Type":"ContainerStarted","Data":"e5da82f028be1cd5547c6b729f8c6ebee675bff495087ca6e9ded5578c0309f6"}
Jul 17 01:10:24.941498 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:24.941278    2573 scope.go:117] "RemoveContainer" containerID="edd8fa43e4302580f3fa6f8359bf3f0e2c73605083bb884810b5f4930802c568"
Jul 17 01:10:24.943490 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:24.943469    2573 scope.go:117] "RemoveContainer" containerID="a6885f91fea363c82f539c9c17e5eb931f0e73772720f1fd296e003856d5851e"
Jul 17 01:10:24.943686 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:24.943654    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl_odh-ai-gateway-infra(1a1859cc-0c24-4d25-a952-6dbdb54624bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd"
Jul 17 01:10:25.951490 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:25.951389    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f55227a-62b8-4163-a235-631cc4790424" containerID="be981f45d379c6f14fd396394e484c9e387e481d2f2dfa6a6d5010ed11d4329e" exitCode=6
Jul 17 01:10:25.951987 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:25.951478    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" event={"ID":"9f55227a-62b8-4163-a235-631cc4790424","Type":"ContainerDied","Data":"be981f45d379c6f14fd396394e484c9e387e481d2f2dfa6a6d5010ed11d4329e"}
Jul 17 01:10:25.951987 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:25.951559    2573 scope.go:117] "RemoveContainer" containerID="edd8fa43e4302580f3fa6f8359bf3f0e2c73605083bb884810b5f4930802c568"
Jul 17 01:10:25.951987 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:25.951782    2573 scope.go:117] "RemoveContainer" containerID="be981f45d379c6f14fd396394e484c9e387e481d2f2dfa6a6d5010ed11d4329e"
Jul 17 01:10:25.952166 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:25.952061    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4_odh-ai-gateway-infra(9f55227a-62b8-4163-a235-631cc4790424)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" podUID="9f55227a-62b8-4163-a235-631cc4790424"
Jul 17 01:10:26.960763 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:26.960729    2573 scope.go:117] "RemoveContainer" containerID="be981f45d379c6f14fd396394e484c9e387e481d2f2dfa6a6d5010ed11d4329e"
Jul 17 01:10:26.961361 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:26.960990    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4_odh-ai-gateway-infra(9f55227a-62b8-4163-a235-631cc4790424)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" podUID="9f55227a-62b8-4163-a235-631cc4790424"
Jul 17 01:10:35.965644 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:35.965598    2573 scope.go:117] "RemoveContainer" containerID="2406b86af96a8dc21622fd11792676006a1cf8621a4d0502921bbf11e1bccb86"
Jul 17 01:10:35.966192 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:35.965769    2573 scope.go:117] "RemoveContainer" containerID="a6885f91fea363c82f539c9c17e5eb931f0e73772720f1fd296e003856d5851e"
Jul 17 01:10:37.026841 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.026773    2573 generic.go:356] "Generic (PLEG): container finished" podID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerID="ce72639b1aa721e69828ce5256c1a3f8a7fda87f6a16884c462d35d7b5b59408" exitCode=6
Jul 17 01:10:37.027354 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.026852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" event={"ID":"1a1859cc-0c24-4d25-a952-6dbdb54624bd","Type":"ContainerDied","Data":"ce72639b1aa721e69828ce5256c1a3f8a7fda87f6a16884c462d35d7b5b59408"}
Jul 17 01:10:37.027354 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.026927    2573 scope.go:117] "RemoveContainer" containerID="a6885f91fea363c82f539c9c17e5eb931f0e73772720f1fd296e003856d5851e"
Jul 17 01:10:37.027354 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.027252    2573 scope.go:117] "RemoveContainer" containerID="ce72639b1aa721e69828ce5256c1a3f8a7fda87f6a16884c462d35d7b5b59408"
Jul 17 01:10:37.027546 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:37.027518    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl_odh-ai-gateway-infra(1a1859cc-0c24-4d25-a952-6dbdb54624bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd"
Jul 17 01:10:37.029108 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.029086    2573 generic.go:356] "Generic (PLEG): container finished" podID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerID="46cc5f088b4ae0a0fbbfd1899a62c977eec8f7e0a893e4fc98ddd748144fd036" exitCode=6
Jul 17 01:10:37.029224 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.029157    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" event={"ID":"83c44d5c-1146-49ec-8720-f3250b3c00db","Type":"ContainerDied","Data":"46cc5f088b4ae0a0fbbfd1899a62c977eec8f7e0a893e4fc98ddd748144fd036"}
Jul 17 01:10:37.029512 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.029495    2573 scope.go:117] "RemoveContainer" containerID="46cc5f088b4ae0a0fbbfd1899a62c977eec8f7e0a893e4fc98ddd748144fd036"
Jul 17 01:10:37.029727 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:37.029705    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4_odh-ai-gateway-infra(83c44d5c-1146-49ec-8720-f3250b3c00db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db"
Jul 17 01:10:37.040247 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.040216    2573 scope.go:117] "RemoveContainer" containerID="2406b86af96a8dc21622fd11792676006a1cf8621a4d0502921bbf11e1bccb86"
Jul 17 01:10:37.963347 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:37.963257    2573 scope.go:117] "RemoveContainer" containerID="be981f45d379c6f14fd396394e484c9e387e481d2f2dfa6a6d5010ed11d4329e"
Jul 17 01:10:38.052620 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.052575    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"]
Jul 17 01:10:38.064676 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.064627    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"]
Jul 17 01:10:38.215575 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.215482    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"
Jul 17 01:10:38.271583 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.271552    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"
Jul 17 01:10:38.332316 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.332275    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1859cc-0c24-4d25-a952-6dbdb54624bd-maas-api-service-ca\") pod \"1a1859cc-0c24-4d25-a952-6dbdb54624bd\" (UID: \"1a1859cc-0c24-4d25-a952-6dbdb54624bd\") "
Jul 17 01:10:38.332515 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.332384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44d5c-1146-49ec-8720-f3250b3c00db-maas-api-service-ca\") pod \"83c44d5c-1146-49ec-8720-f3250b3c00db\" (UID: \"83c44d5c-1146-49ec-8720-f3250b3c00db\") "
Jul 17 01:10:38.332774 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.332746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1859cc-0c24-4d25-a952-6dbdb54624bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a1859cc-0c24-4d25-a952-6dbdb54624bd" (UID: "1a1859cc-0c24-4d25-a952-6dbdb54624bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:10:38.332856 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.332746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c44d5c-1146-49ec-8720-f3250b3c00db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83c44d5c-1146-49ec-8720-f3250b3c00db" (UID: "83c44d5c-1146-49ec-8720-f3250b3c00db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:10:38.433158 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.433116    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44d5c-1146-49ec-8720-f3250b3c00db-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:10:38.433158 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:38.433149    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1859cc-0c24-4d25-a952-6dbdb54624bd-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:10:39.049767 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.049719    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f55227a-62b8-4163-a235-631cc4790424" containerID="c42f2f8145559037649a434b480446b26b91f8016bde8af45d41b8786bf557c4" exitCode=6
Jul 17 01:10:39.050007 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.049808    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" event={"ID":"9f55227a-62b8-4163-a235-631cc4790424","Type":"ContainerDied","Data":"c42f2f8145559037649a434b480446b26b91f8016bde8af45d41b8786bf557c4"}
Jul 17 01:10:39.050007 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.049880    2573 scope.go:117] "RemoveContainer" containerID="be981f45d379c6f14fd396394e484c9e387e481d2f2dfa6a6d5010ed11d4329e"
Jul 17 01:10:39.050237 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.050214    2573 scope.go:117] "RemoveContainer" containerID="c42f2f8145559037649a434b480446b26b91f8016bde8af45d41b8786bf557c4"
Jul 17 01:10:39.050528 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:10:39.050501    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4_odh-ai-gateway-infra(9f55227a-62b8-4163-a235-631cc4790424)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" podUID="9f55227a-62b8-4163-a235-631cc4790424"
Jul 17 01:10:39.051677 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.051658    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"
Jul 17 01:10:39.051816 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.051695    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl" event={"ID":"1a1859cc-0c24-4d25-a952-6dbdb54624bd","Type":"ContainerDied","Data":"8f56cebda54a94fa4e29b73822add5e05280d146f82c5997dba377a169abbe63"}
Jul 17 01:10:39.053413 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.053387    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"
Jul 17 01:10:39.053841 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.053437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4" event={"ID":"83c44d5c-1146-49ec-8720-f3250b3c00db","Type":"ContainerDied","Data":"06315edb13d0feee553e51acc024723eb11859b6ea5be17b1c39124360c3f203"}
Jul 17 01:10:39.065092 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.064913    2573 scope.go:117] "RemoveContainer" containerID="ce72639b1aa721e69828ce5256c1a3f8a7fda87f6a16884c462d35d7b5b59408"
Jul 17 01:10:39.075244 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.075219    2573 scope.go:117] "RemoveContainer" containerID="46cc5f088b4ae0a0fbbfd1899a62c977eec8f7e0a893e4fc98ddd748144fd036"
Jul 17 01:10:39.076173 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.076137    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"]
Jul 17 01:10:39.079549 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.079517    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-mflfl"]
Jul 17 01:10:39.090126 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.090087    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"]
Jul 17 01:10:39.094352 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.094316    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-nl6k4"]
Jul 17 01:10:39.968926 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.968885    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" path="/var/lib/kubelet/pods/1a1859cc-0c24-4d25-a952-6dbdb54624bd/volumes"
Jul 17 01:10:39.969265 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:39.969251    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" path="/var/lib/kubelet/pods/83c44d5c-1146-49ec-8720-f3250b3c00db/volumes"
Jul 17 01:10:40.074191 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:40.074146    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"]
Jul 17 01:10:40.223178 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:40.223090    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"
Jul 17 01:10:40.251769 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:40.251729    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f55227a-62b8-4163-a235-631cc4790424-maas-api-service-ca\") pod \"9f55227a-62b8-4163-a235-631cc4790424\" (UID: \"9f55227a-62b8-4163-a235-631cc4790424\") "
Jul 17 01:10:40.252131 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:40.252108    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f55227a-62b8-4163-a235-631cc4790424-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f55227a-62b8-4163-a235-631cc4790424" (UID: "9f55227a-62b8-4163-a235-631cc4790424"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:10:40.353315 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:40.353279    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f55227a-62b8-4163-a235-631cc4790424-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:10:41.071006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:41.070974    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"
Jul 17 01:10:41.071184 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:41.071017    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4" event={"ID":"9f55227a-62b8-4163-a235-631cc4790424","Type":"ContainerDied","Data":"e5da82f028be1cd5547c6b729f8c6ebee675bff495087ca6e9ded5578c0309f6"}
Jul 17 01:10:41.071184 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:41.071082    2573 scope.go:117] "RemoveContainer" containerID="c42f2f8145559037649a434b480446b26b91f8016bde8af45d41b8786bf557c4"
Jul 17 01:10:41.095263 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:41.095223    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"]
Jul 17 01:10:41.099206 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:41.099171    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-g4kq4"]
Jul 17 01:10:41.970858 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:10:41.970808    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f55227a-62b8-4163-a235-631cc4790424" path="/var/lib/kubelet/pods/9f55227a-62b8-4163-a235-631cc4790424/volumes"
Jul 17 01:11:09.413690 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.413658    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"]
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414082    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414095    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414104    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414109    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414119    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414125    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414132    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414137    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414144    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414150    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414158    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414163    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414169    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414174    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414184    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414189    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414243    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414250    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414255    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414261    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414268    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414273    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414278    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.414305 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.414284    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f55227a-62b8-4163-a235-631cc4790424" containerName="revoke-keys"
Jul 17 01:11:09.417479 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.417459    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"
Jul 17 01:11:09.419154 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.419133    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:11:09.423742 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.423718    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"]
Jul 17 01:11:09.496811 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.496763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b734e8b1-1571-44c6-b17f-cc2da18e471b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-7247k\" (UID: \"b734e8b1-1571-44c6-b17f-cc2da18e471b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"
Jul 17 01:11:09.597527 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.597499    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b734e8b1-1571-44c6-b17f-cc2da18e471b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-7247k\" (UID: \"b734e8b1-1571-44c6-b17f-cc2da18e471b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"
Jul 17 01:11:09.598127 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.598107    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b734e8b1-1571-44c6-b17f-cc2da18e471b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-7247k\" (UID: \"b734e8b1-1571-44c6-b17f-cc2da18e471b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"
Jul 17 01:11:09.728186 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.728128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"
Jul 17 01:11:09.858687 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:09.858660    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"]
Jul 17 01:11:09.859889 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:11:09.859847    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb734e8b1_1571_44c6_b17f_cc2da18e471b.slice/crio-0c29fc46b28e32790d41b795fbf4adf83f033dd095859f8701f70194d99a3b8a WatchSource:0}: Error finding container 0c29fc46b28e32790d41b795fbf4adf83f033dd095859f8701f70194d99a3b8a: Status 404 returned error can't find the container with id 0c29fc46b28e32790d41b795fbf4adf83f033dd095859f8701f70194d99a3b8a
Jul 17 01:11:10.235051 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:10.235021    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" event={"ID":"b734e8b1-1571-44c6-b17f-cc2da18e471b","Type":"ContainerStarted","Data":"2f9316b250eb97137478c47e893dba9f86e3398cd94970795e27ece2dae58d50"}
Jul 17 01:11:10.235194 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:10.235085    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" event={"ID":"b734e8b1-1571-44c6-b17f-cc2da18e471b","Type":"ContainerStarted","Data":"0c29fc46b28e32790d41b795fbf4adf83f033dd095859f8701f70194d99a3b8a"}
Jul 17 01:11:10.250344 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:10.250305    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" podStartSLOduration=1.250290867 podStartE2EDuration="1.250290867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:11:10.248435042 +0000 UTC m=+3264.824587619" watchObservedRunningTime="2026-07-17 01:11:10.250290867 +0000 UTC m=+3264.826443441"
Jul 17 01:11:11.247299 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.247255    2573 generic.go:356] "Generic (PLEG): container finished" podID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerID="2f9316b250eb97137478c47e893dba9f86e3398cd94970795e27ece2dae58d50" exitCode=6
Jul 17 01:11:11.247829 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.247339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" event={"ID":"b734e8b1-1571-44c6-b17f-cc2da18e471b","Type":"ContainerDied","Data":"2f9316b250eb97137478c47e893dba9f86e3398cd94970795e27ece2dae58d50"}
Jul 17 01:11:11.247829 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.247569    2573 scope.go:117] "RemoveContainer" containerID="2f9316b250eb97137478c47e893dba9f86e3398cd94970795e27ece2dae58d50"
Jul 17 01:11:11.267016 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.266991    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"]
Jul 17 01:11:11.267429 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.267411    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:11.267429 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.267427    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1859cc-0c24-4d25-a952-6dbdb54624bd" containerName="revoke-keys"
Jul 17 01:11:11.267543 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.267489    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c44d5c-1146-49ec-8720-f3250b3c00db" containerName="revoke-keys"
Jul 17 01:11:11.271651 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.271630    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"
Jul 17 01:11:11.276819 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.276775    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"]
Jul 17 01:11:11.412425 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.412390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d615a4-513f-438b-867c-a36d902b23ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb\" (UID: \"24d615a4-513f-438b-867c-a36d902b23ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"
Jul 17 01:11:11.512929 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.512887    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d615a4-513f-438b-867c-a36d902b23ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb\" (UID: \"24d615a4-513f-438b-867c-a36d902b23ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"
Jul 17 01:11:11.513547 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.513521    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d615a4-513f-438b-867c-a36d902b23ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb\" (UID: \"24d615a4-513f-438b-867c-a36d902b23ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"
Jul 17 01:11:11.592994 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.592962    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"
Jul 17 01:11:11.728269 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:11.728236    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"]
Jul 17 01:11:11.729651 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:11:11.729614    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d615a4_513f_438b_867c_a36d902b23ea.slice/crio-b15a1b58c2395e909efdf4cffd90de24ff7c6275a720f7b1f006f069e578fac8 WatchSource:0}: Error finding container b15a1b58c2395e909efdf4cffd90de24ff7c6275a720f7b1f006f069e578fac8: Status 404 returned error can't find the container with id b15a1b58c2395e909efdf4cffd90de24ff7c6275a720f7b1f006f069e578fac8
Jul 17 01:11:12.256117 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:12.256082    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" event={"ID":"24d615a4-513f-438b-867c-a36d902b23ea","Type":"ContainerStarted","Data":"4232881dea8187e349b8abeaf2267e63ec4febb29c578c80c55709c210e42c74"}
Jul 17 01:11:12.256626 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:12.256142    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" event={"ID":"24d615a4-513f-438b-867c-a36d902b23ea","Type":"ContainerStarted","Data":"b15a1b58c2395e909efdf4cffd90de24ff7c6275a720f7b1f006f069e578fac8"}
Jul 17 01:11:12.258176 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:12.258149    2573 generic.go:356] "Generic (PLEG): container finished" podID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerID="baf1a7ff695ba9deb6b6f4819e0f55c2fd4979f05574601e0c898c845bdf3d0f" exitCode=6
Jul 17 01:11:12.258323 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:12.258218    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" event={"ID":"b734e8b1-1571-44c6-b17f-cc2da18e471b","Type":"ContainerDied","Data":"baf1a7ff695ba9deb6b6f4819e0f55c2fd4979f05574601e0c898c845bdf3d0f"}
Jul 17 01:11:12.258323 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:12.258282    2573 scope.go:117] "RemoveContainer" containerID="2f9316b250eb97137478c47e893dba9f86e3398cd94970795e27ece2dae58d50"
Jul 17 01:11:12.258477 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:12.258452    2573 scope.go:117] "RemoveContainer" containerID="baf1a7ff695ba9deb6b6f4819e0f55c2fd4979f05574601e0c898c845bdf3d0f"
Jul 17 01:11:12.258703 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:12.258683    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-7247k_odh-ai-gateway-infra(b734e8b1-1571-44c6-b17f-cc2da18e471b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b"
Jul 17 01:11:12.271045 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:12.270986    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" podStartSLOduration=1.27096675 podStartE2EDuration="1.27096675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:11:12.267882992 +0000 UTC m=+3266.844035567" watchObservedRunningTime="2026-07-17 01:11:12.27096675 +0000 UTC m=+3266.847119387"
Jul 17 01:11:13.265455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.265420    2573 generic.go:356] "Generic (PLEG): container finished" podID="24d615a4-513f-438b-867c-a36d902b23ea" containerID="4232881dea8187e349b8abeaf2267e63ec4febb29c578c80c55709c210e42c74" exitCode=6
Jul 17 01:11:13.266013 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.265512    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" event={"ID":"24d615a4-513f-438b-867c-a36d902b23ea","Type":"ContainerDied","Data":"4232881dea8187e349b8abeaf2267e63ec4febb29c578c80c55709c210e42c74"}
Jul 17 01:11:13.266013 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.265720    2573 scope.go:117] "RemoveContainer" containerID="4232881dea8187e349b8abeaf2267e63ec4febb29c578c80c55709c210e42c74"
Jul 17 01:11:13.267537 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.267521    2573 scope.go:117] "RemoveContainer" containerID="baf1a7ff695ba9deb6b6f4819e0f55c2fd4979f05574601e0c898c845bdf3d0f"
Jul 17 01:11:13.267695 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:13.267676    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-7247k_odh-ai-gateway-infra(b734e8b1-1571-44c6-b17f-cc2da18e471b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b"
Jul 17 01:11:13.691576 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.691492    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"]
Jul 17 01:11:13.694976 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.694959    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"
Jul 17 01:11:13.701103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.701077    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"]
Jul 17 01:11:13.833018 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.832987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt\" (UID: \"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"
Jul 17 01:11:13.933930 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.933896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt\" (UID: \"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"
Jul 17 01:11:13.934511 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:13.934492    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt\" (UID: \"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"
Jul 17 01:11:14.007089 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:14.007064    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"
Jul 17 01:11:14.136129 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:14.136096    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"]
Jul 17 01:11:14.137849 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:11:14.137821    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af40ce9_6ee8_4d0c_be01_0e0ba1e3f2f2.slice/crio-d6df3a1013c38a6a4511c6a4e94b42ec5c82fd680b3221301ef16997b479eabc WatchSource:0}: Error finding container d6df3a1013c38a6a4511c6a4e94b42ec5c82fd680b3221301ef16997b479eabc: Status 404 returned error can't find the container with id d6df3a1013c38a6a4511c6a4e94b42ec5c82fd680b3221301ef16997b479eabc
Jul 17 01:11:14.274233 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:14.274203    2573 generic.go:356] "Generic (PLEG): container finished" podID="24d615a4-513f-438b-867c-a36d902b23ea" containerID="90ffd909646be26cc42dea3a9bc8283c7c88a65b0d2650c91f1d78826b5e697f" exitCode=6
Jul 17 01:11:14.274635 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:14.274288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" event={"ID":"24d615a4-513f-438b-867c-a36d902b23ea","Type":"ContainerDied","Data":"90ffd909646be26cc42dea3a9bc8283c7c88a65b0d2650c91f1d78826b5e697f"}
Jul 17 01:11:14.274635 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:14.274363    2573 scope.go:117] "RemoveContainer" containerID="4232881dea8187e349b8abeaf2267e63ec4febb29c578c80c55709c210e42c74"
Jul 17 01:11:14.274635 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:14.274475    2573 scope.go:117] "RemoveContainer" containerID="90ffd909646be26cc42dea3a9bc8283c7c88a65b0d2650c91f1d78826b5e697f"
Jul 17 01:11:14.274780 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:14.274717    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb_odh-ai-gateway-infra(24d615a4-513f-438b-867c-a36d902b23ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" podUID="24d615a4-513f-438b-867c-a36d902b23ea"
Jul 17 01:11:14.275627 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:14.275606    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" event={"ID":"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2","Type":"ContainerStarted","Data":"d6df3a1013c38a6a4511c6a4e94b42ec5c82fd680b3221301ef16997b479eabc"}
Jul 17 01:11:15.283478 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:15.283447    2573 scope.go:117] "RemoveContainer" containerID="90ffd909646be26cc42dea3a9bc8283c7c88a65b0d2650c91f1d78826b5e697f"
Jul 17 01:11:15.283993 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:15.283725    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb_odh-ai-gateway-infra(24d615a4-513f-438b-867c-a36d902b23ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" podUID="24d615a4-513f-438b-867c-a36d902b23ea"
Jul 17 01:11:15.284904 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:15.284877    2573 generic.go:356] "Generic (PLEG): container finished" podID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerID="c5419ec02def53d3473bc40e467cf591d3eb531c23c1b435e065e314dee5fa3c" exitCode=6
Jul 17 01:11:15.285013 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:15.284964    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" event={"ID":"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2","Type":"ContainerDied","Data":"c5419ec02def53d3473bc40e467cf591d3eb531c23c1b435e065e314dee5fa3c"}
Jul 17 01:11:15.285156 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:15.285142    2573 scope.go:117] "RemoveContainer" containerID="c5419ec02def53d3473bc40e467cf591d3eb531c23c1b435e065e314dee5fa3c"
Jul 17 01:11:16.292669 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:16.292631    2573 generic.go:356] "Generic (PLEG): container finished" podID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerID="89d8c85f7d222dace34dfddaf5cb2e209b9149d6e174feeda4f713d85c7db272" exitCode=6
Jul 17 01:11:16.293172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:16.292714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" event={"ID":"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2","Type":"ContainerDied","Data":"89d8c85f7d222dace34dfddaf5cb2e209b9149d6e174feeda4f713d85c7db272"}
Jul 17 01:11:16.293172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:16.292776    2573 scope.go:117] "RemoveContainer" containerID="c5419ec02def53d3473bc40e467cf591d3eb531c23c1b435e065e314dee5fa3c"
Jul 17 01:11:16.293172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:16.293014    2573 scope.go:117] "RemoveContainer" containerID="89d8c85f7d222dace34dfddaf5cb2e209b9149d6e174feeda4f713d85c7db272"
Jul 17 01:11:16.293682 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:16.293210    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt_odh-ai-gateway-infra(9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2"
Jul 17 01:11:17.300640 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:17.300615    2573 scope.go:117] "RemoveContainer" containerID="89d8c85f7d222dace34dfddaf5cb2e209b9149d6e174feeda4f713d85c7db272"
Jul 17 01:11:17.301075 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:17.300817    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt_odh-ai-gateway-infra(9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2"
Jul 17 01:11:23.962643 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:23.962616    2573 scope.go:117] "RemoveContainer" containerID="baf1a7ff695ba9deb6b6f4819e0f55c2fd4979f05574601e0c898c845bdf3d0f"
Jul 17 01:11:24.340993 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:24.340953    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" event={"ID":"b734e8b1-1571-44c6-b17f-cc2da18e471b","Type":"ContainerStarted","Data":"be59c9c3860d2bd3bcd119bb164633ccabe462ca8138c5825e21e91df3770fa3"}
Jul 17 01:11:25.348568 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:25.348534    2573 generic.go:356] "Generic (PLEG): container finished" podID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerID="be59c9c3860d2bd3bcd119bb164633ccabe462ca8138c5825e21e91df3770fa3" exitCode=6
Jul 17 01:11:25.349007 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:25.348605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" event={"ID":"b734e8b1-1571-44c6-b17f-cc2da18e471b","Type":"ContainerDied","Data":"be59c9c3860d2bd3bcd119bb164633ccabe462ca8138c5825e21e91df3770fa3"}
Jul 17 01:11:25.349007 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:25.348664    2573 scope.go:117] "RemoveContainer" containerID="baf1a7ff695ba9deb6b6f4819e0f55c2fd4979f05574601e0c898c845bdf3d0f"
Jul 17 01:11:25.352999 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:25.349264    2573 scope.go:117] "RemoveContainer" containerID="be59c9c3860d2bd3bcd119bb164633ccabe462ca8138c5825e21e91df3770fa3"
Jul 17 01:11:25.352999 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:25.349679    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-7247k_odh-ai-gateway-infra(b734e8b1-1571-44c6-b17f-cc2da18e471b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b"
Jul 17 01:11:25.365421 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:25.365401    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"]
Jul 17 01:11:26.490460 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:26.490431    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"
Jul 17 01:11:26.639269 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:26.639186    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b734e8b1-1571-44c6-b17f-cc2da18e471b-maas-api-service-ca\") pod \"b734e8b1-1571-44c6-b17f-cc2da18e471b\" (UID: \"b734e8b1-1571-44c6-b17f-cc2da18e471b\") "
Jul 17 01:11:26.639590 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:26.639561    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b734e8b1-1571-44c6-b17f-cc2da18e471b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b734e8b1-1571-44c6-b17f-cc2da18e471b" (UID: "b734e8b1-1571-44c6-b17f-cc2da18e471b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:11:26.740690 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:26.740662    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b734e8b1-1571-44c6-b17f-cc2da18e471b-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:11:26.963148 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:26.963068    2573 scope.go:117] "RemoveContainer" containerID="90ffd909646be26cc42dea3a9bc8283c7c88a65b0d2650c91f1d78826b5e697f"
Jul 17 01:11:27.363726 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:27.363703    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"
Jul 17 01:11:27.363907 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:27.363749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k" event={"ID":"b734e8b1-1571-44c6-b17f-cc2da18e471b","Type":"ContainerDied","Data":"0c29fc46b28e32790d41b795fbf4adf83f033dd095859f8701f70194d99a3b8a"}
Jul 17 01:11:27.363907 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:27.363828    2573 scope.go:117] "RemoveContainer" containerID="be59c9c3860d2bd3bcd119bb164633ccabe462ca8138c5825e21e91df3770fa3"
Jul 17 01:11:27.365952 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:27.365928    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" event={"ID":"24d615a4-513f-438b-867c-a36d902b23ea","Type":"ContainerStarted","Data":"3de85398b4019a263f11b0d22a51c7fc8d8c1a958282fc13eb399318313ddd13"}
Jul 17 01:11:27.392779 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:27.392756    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"]
Jul 17 01:11:27.397918 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:27.397897    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-7247k"]
Jul 17 01:11:27.968467 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:27.968393    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" path="/var/lib/kubelet/pods/b734e8b1-1571-44c6-b17f-cc2da18e471b/volumes"
Jul 17 01:11:28.373767 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:28.373734    2573 generic.go:356] "Generic (PLEG): container finished" podID="24d615a4-513f-438b-867c-a36d902b23ea" containerID="3de85398b4019a263f11b0d22a51c7fc8d8c1a958282fc13eb399318313ddd13" exitCode=6
Jul 17 01:11:28.374004 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:28.373830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" event={"ID":"24d615a4-513f-438b-867c-a36d902b23ea","Type":"ContainerDied","Data":"3de85398b4019a263f11b0d22a51c7fc8d8c1a958282fc13eb399318313ddd13"}
Jul 17 01:11:28.374004 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:28.373906    2573 scope.go:117] "RemoveContainer" containerID="90ffd909646be26cc42dea3a9bc8283c7c88a65b0d2650c91f1d78826b5e697f"
Jul 17 01:11:28.374178 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:28.374154    2573 scope.go:117] "RemoveContainer" containerID="3de85398b4019a263f11b0d22a51c7fc8d8c1a958282fc13eb399318313ddd13"
Jul 17 01:11:28.374438 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:28.374408    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb_odh-ai-gateway-infra(24d615a4-513f-438b-867c-a36d902b23ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" podUID="24d615a4-513f-438b-867c-a36d902b23ea"
Jul 17 01:11:28.389698 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:28.389675    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"]
Jul 17 01:11:28.962700 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:28.962668    2573 scope.go:117] "RemoveContainer" containerID="89d8c85f7d222dace34dfddaf5cb2e209b9149d6e174feeda4f713d85c7db272"
Jul 17 01:11:29.384855 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:29.384830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" event={"ID":"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2","Type":"ContainerStarted","Data":"746cea9efd7ce3ddb2885575ed1657652e0c196dc3806d82536d3e4613ced837"}
Jul 17 01:11:29.410633 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:29.410584    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" podStartSLOduration=16.410568824 podStartE2EDuration="16.410568824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:11:29.40816042 +0000 UTC m=+3283.984312996" watchObservedRunningTime="2026-07-17 01:11:29.410568824 +0000 UTC m=+3283.986721399"
Jul 17 01:11:29.520898 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:29.520877    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"
Jul 17 01:11:29.564172 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:29.564144    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d615a4-513f-438b-867c-a36d902b23ea-maas-api-service-ca\") pod \"24d615a4-513f-438b-867c-a36d902b23ea\" (UID: \"24d615a4-513f-438b-867c-a36d902b23ea\") "
Jul 17 01:11:29.564445 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:29.564424    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d615a4-513f-438b-867c-a36d902b23ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24d615a4-513f-438b-867c-a36d902b23ea" (UID: "24d615a4-513f-438b-867c-a36d902b23ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:11:29.665081 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:29.665058    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d615a4-513f-438b-867c-a36d902b23ea-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:11:30.392469 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.392442    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"
Jul 17 01:11:30.392947 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.392466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb" event={"ID":"24d615a4-513f-438b-867c-a36d902b23ea","Type":"ContainerDied","Data":"b15a1b58c2395e909efdf4cffd90de24ff7c6275a720f7b1f006f069e578fac8"}
Jul 17 01:11:30.392947 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.392536    2573 scope.go:117] "RemoveContainer" containerID="3de85398b4019a263f11b0d22a51c7fc8d8c1a958282fc13eb399318313ddd13"
Jul 17 01:11:30.394716 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.394689    2573 generic.go:356] "Generic (PLEG): container finished" podID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerID="746cea9efd7ce3ddb2885575ed1657652e0c196dc3806d82536d3e4613ced837" exitCode=6
Jul 17 01:11:30.394885 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.394760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" event={"ID":"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2","Type":"ContainerDied","Data":"746cea9efd7ce3ddb2885575ed1657652e0c196dc3806d82536d3e4613ced837"}
Jul 17 01:11:30.395131 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.395115    2573 scope.go:117] "RemoveContainer" containerID="746cea9efd7ce3ddb2885575ed1657652e0c196dc3806d82536d3e4613ced837"
Jul 17 01:11:30.395349 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:30.395327    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt_odh-ai-gateway-infra(9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2"
Jul 17 01:11:30.401506 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.401491    2573 scope.go:117] "RemoveContainer" containerID="89d8c85f7d222dace34dfddaf5cb2e209b9149d6e174feeda4f713d85c7db272"
Jul 17 01:11:30.407429 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.407407    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"]
Jul 17 01:11:30.411108 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.411086    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6dqhb"]
Jul 17 01:11:30.420567 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:30.420547    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"]
Jul 17 01:11:31.541503 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:31.541475    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"
Jul 17 01:11:31.581949 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:31.581926    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2-maas-api-service-ca\") pod \"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2\" (UID: \"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2\") "
Jul 17 01:11:31.585812 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:31.582633    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" (UID: "9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:11:31.683193 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:31.683137    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:11:31.968699 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:31.968638    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d615a4-513f-438b-867c-a36d902b23ea" path="/var/lib/kubelet/pods/24d615a4-513f-438b-867c-a36d902b23ea/volumes"
Jul 17 01:11:32.410567 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:32.410542    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"
Jul 17 01:11:32.410743 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:32.410536    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt" event={"ID":"9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2","Type":"ContainerDied","Data":"d6df3a1013c38a6a4511c6a4e94b42ec5c82fd680b3221301ef16997b479eabc"}
Jul 17 01:11:32.410743 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:32.410669    2573 scope.go:117] "RemoveContainer" containerID="746cea9efd7ce3ddb2885575ed1657652e0c196dc3806d82536d3e4613ced837"
Jul 17 01:11:32.424733 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:32.424711    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"]
Jul 17 01:11:32.428218 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:32.428196    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7c9gt"]
Jul 17 01:11:33.968091 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:33.968060    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" path="/var/lib/kubelet/pods/9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2/volumes"
Jul 17 01:11:36.731527 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.731498    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"]
Jul 17 01:11:36.731994 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.731969    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:36.731994 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.731992    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732003    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732011    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732018    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732024    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732033    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732038    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732044    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732049    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732070    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732077    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732086    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732091    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732096    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732101    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732166    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732174    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:36.732175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732181    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.732708 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732188    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:36.732708 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732195    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:36.732708 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.732201    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:36.744688 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.744563    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"]
Jul 17 01:11:36.746422 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.744667    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"
Jul 17 01:11:36.748004 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.747984    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:11:36.821837 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.821806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c98e8f-b78d-48f5-967b-de9f3cf84e6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-6h9vb\" (UID: \"26c98e8f-b78d-48f5-967b-de9f3cf84e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"
Jul 17 01:11:36.923133 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.923105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c98e8f-b78d-48f5-967b-de9f3cf84e6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-6h9vb\" (UID: \"26c98e8f-b78d-48f5-967b-de9f3cf84e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"
Jul 17 01:11:36.923668 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:36.923648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c98e8f-b78d-48f5-967b-de9f3cf84e6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-6h9vb\" (UID: \"26c98e8f-b78d-48f5-967b-de9f3cf84e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"
Jul 17 01:11:37.057214 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:37.057189    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"
Jul 17 01:11:37.188719 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:37.188691    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"]
Jul 17 01:11:37.190539 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:11:37.190510    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c98e8f_b78d_48f5_967b_de9f3cf84e6a.slice/crio-5d0ba549566e9e10948ca3a40727dc17ae4e0ed1fce106bb9e9d80552820f647 WatchSource:0}: Error finding container 5d0ba549566e9e10948ca3a40727dc17ae4e0ed1fce106bb9e9d80552820f647: Status 404 returned error can't find the container with id 5d0ba549566e9e10948ca3a40727dc17ae4e0ed1fce106bb9e9d80552820f647
Jul 17 01:11:37.441872 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:37.441821    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" event={"ID":"26c98e8f-b78d-48f5-967b-de9f3cf84e6a","Type":"ContainerStarted","Data":"5d0ba549566e9e10948ca3a40727dc17ae4e0ed1fce106bb9e9d80552820f647"}
Jul 17 01:11:38.449011 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:38.448979    2573 generic.go:356] "Generic (PLEG): container finished" podID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerID="dbc836671dad9e41611dbec8baa79f9b785851bf67eaee396179952256d9166b" exitCode=6
Jul 17 01:11:38.449413 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:38.449024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" event={"ID":"26c98e8f-b78d-48f5-967b-de9f3cf84e6a","Type":"ContainerDied","Data":"dbc836671dad9e41611dbec8baa79f9b785851bf67eaee396179952256d9166b"}
Jul 17 01:11:38.449413 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:38.449189    2573 scope.go:117] "RemoveContainer" containerID="dbc836671dad9e41611dbec8baa79f9b785851bf67eaee396179952256d9166b"
Jul 17 01:11:39.457131 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:39.457046    2573 generic.go:356] "Generic (PLEG): container finished" podID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerID="6c32368060ee110222b3d25efd4ef741dcb91d46e3fea31d7f8d551cc53b3298" exitCode=6
Jul 17 01:11:39.457131 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:39.457110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" event={"ID":"26c98e8f-b78d-48f5-967b-de9f3cf84e6a","Type":"ContainerDied","Data":"6c32368060ee110222b3d25efd4ef741dcb91d46e3fea31d7f8d551cc53b3298"}
Jul 17 01:11:39.457585 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:39.457173    2573 scope.go:117] "RemoveContainer" containerID="dbc836671dad9e41611dbec8baa79f9b785851bf67eaee396179952256d9166b"
Jul 17 01:11:39.457585 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:39.457341    2573 scope.go:117] "RemoveContainer" containerID="6c32368060ee110222b3d25efd4ef741dcb91d46e3fea31d7f8d551cc53b3298"
Jul 17 01:11:39.457699 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:39.457579    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-6h9vb_odh-ai-gateway-infra(26c98e8f-b78d-48f5-967b-de9f3cf84e6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a"
Jul 17 01:11:40.465225 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:40.465199    2573 scope.go:117] "RemoveContainer" containerID="6c32368060ee110222b3d25efd4ef741dcb91d46e3fea31d7f8d551cc53b3298"
Jul 17 01:11:40.465596 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:40.465361    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-6h9vb_odh-ai-gateway-infra(26c98e8f-b78d-48f5-967b-de9f3cf84e6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a"
Jul 17 01:11:44.353014 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.352977    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7469-139661443547760"]
Jul 17 01:11:44.353455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.353355    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:44.353455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.353366    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:44.353455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.353425    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d615a4-513f-438b-867c-a36d902b23ea" containerName="revoke-keys"
Jul 17 01:11:44.353455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.353431    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b734e8b1-1571-44c6-b17f-cc2da18e471b" containerName="revoke-keys"
Jul 17 01:11:44.353455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.353440    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af40ce9-6ee8-4d0c-be01-0e0ba1e3f2f2" containerName="revoke-keys"
Jul 17 01:11:44.357654 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.357618    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760"
Jul 17 01:11:44.359692 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.359670    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czq5d\""
Jul 17 01:11:44.362264 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.362236    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"]
Jul 17 01:11:44.485566 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.485540    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7kf\" (UniqueName: \"kubernetes.io/projected/e170e529-2011-49a7-870b-26e893e82b39-kube-api-access-2z7kf\") pod \"test-curl-7469-139661443547760\" (UID: \"e170e529-2011-49a7-870b-26e893e82b39\") " pod="opendatahub/test-curl-7469-139661443547760"
Jul 17 01:11:44.586117 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.586089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7kf\" (UniqueName: \"kubernetes.io/projected/e170e529-2011-49a7-870b-26e893e82b39-kube-api-access-2z7kf\") pod \"test-curl-7469-139661443547760\" (UID: \"e170e529-2011-49a7-870b-26e893e82b39\") " pod="opendatahub/test-curl-7469-139661443547760"
Jul 17 01:11:44.593667 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.593638    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7kf\" (UniqueName: \"kubernetes.io/projected/e170e529-2011-49a7-870b-26e893e82b39-kube-api-access-2z7kf\") pod \"test-curl-7469-139661443547760\" (UID: \"e170e529-2011-49a7-870b-26e893e82b39\") " pod="opendatahub/test-curl-7469-139661443547760"
Jul 17 01:11:44.669649 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.669592    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760"
Jul 17 01:11:44.798358 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.798339    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"]
Jul 17 01:11:44.800680 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:11:44.800655    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode170e529_2011_49a7_870b_26e893e82b39.slice/crio-342a687a7ed686d5e946cd32e767c4ca272c3d9e997b18f5eb2ab1082cbd29c6 WatchSource:0}: Error finding container 342a687a7ed686d5e946cd32e767c4ca272c3d9e997b18f5eb2ab1082cbd29c6: Status 404 returned error can't find the container with id 342a687a7ed686d5e946cd32e767c4ca272c3d9e997b18f5eb2ab1082cbd29c6
Jul 17 01:11:44.801928 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:44.801911    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 01:11:45.497671 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:45.497635    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443547760" event={"ID":"e170e529-2011-49a7-870b-26e893e82b39","Type":"ContainerStarted","Data":"342a687a7ed686d5e946cd32e767c4ca272c3d9e997b18f5eb2ab1082cbd29c6"}
Jul 17 01:11:46.504556 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:46.504520    2573 generic.go:356] "Generic (PLEG): container finished" podID="e170e529-2011-49a7-870b-26e893e82b39" containerID="fcb794962243a43a149140add9238c178281d5d8e66cd6e2daaa4e10eed7d9b6" exitCode=0
Jul 17 01:11:46.505005 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:46.504602    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443547760" event={"ID":"e170e529-2011-49a7-870b-26e893e82b39","Type":"ContainerDied","Data":"fcb794962243a43a149140add9238c178281d5d8e66cd6e2daaa4e10eed7d9b6"}
Jul 17 01:11:47.648018 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.647994    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760"
Jul 17 01:11:47.679638 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.679605    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7469-139661443547760_e170e529-2011-49a7-870b-26e893e82b39/test-curl-7469-139661443547760/0.log"
Jul 17 01:11:47.701446 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.701420    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"]
Jul 17 01:11:47.703830 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.703808    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"]
Jul 17 01:11:47.712374 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.712353    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z7kf\" (UniqueName: \"kubernetes.io/projected/e170e529-2011-49a7-870b-26e893e82b39-kube-api-access-2z7kf\") pod \"e170e529-2011-49a7-870b-26e893e82b39\" (UID: \"e170e529-2011-49a7-870b-26e893e82b39\") "
Jul 17 01:11:47.714342 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.714318    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e170e529-2011-49a7-870b-26e893e82b39-kube-api-access-2z7kf" (OuterVolumeSpecName: "kube-api-access-2z7kf") pod "e170e529-2011-49a7-870b-26e893e82b39" (UID: "e170e529-2011-49a7-870b-26e893e82b39"). InnerVolumeSpecName "kube-api-access-2z7kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:11:47.813586 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.813537    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7kf\" (UniqueName: \"kubernetes.io/projected/e170e529-2011-49a7-870b-26e893e82b39-kube-api-access-2z7kf\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:11:47.968770 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.968666    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e170e529-2011-49a7-870b-26e893e82b39" path="/var/lib/kubelet/pods/e170e529-2011-49a7-870b-26e893e82b39/volumes"
Jul 17 01:11:48.001006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.977941    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 01:11:48.001006 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:47.991737    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 01:11:48.516876 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:48.516847    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760"
Jul 17 01:11:48.517063 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:48.516855    2573 scope.go:117] "RemoveContainer" containerID="fcb794962243a43a149140add9238c178281d5d8e66cd6e2daaa4e10eed7d9b6"
Jul 17 01:11:51.962515 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:51.962487    2573 scope.go:117] "RemoveContainer" containerID="6c32368060ee110222b3d25efd4ef741dcb91d46e3fea31d7f8d551cc53b3298"
Jul 17 01:11:52.542456 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:52.542425    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" event={"ID":"26c98e8f-b78d-48f5-967b-de9f3cf84e6a","Type":"ContainerStarted","Data":"e5f9d5a08849014d565612bc13a01a69e0f37ee43ce820a89797f3f1ee5709fe"}
Jul 17 01:11:52.557313 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:52.557270    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" podStartSLOduration=16.557257255 podStartE2EDuration="16.557257255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:11:52.555097995 +0000 UTC m=+3307.131250569" watchObservedRunningTime="2026-07-17 01:11:52.557257255 +0000 UTC m=+3307.133409829"
Jul 17 01:11:53.550463 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:53.550424    2573 generic.go:356] "Generic (PLEG): container finished" podID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerID="e5f9d5a08849014d565612bc13a01a69e0f37ee43ce820a89797f3f1ee5709fe" exitCode=6
Jul 17 01:11:53.550926 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:53.550471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" event={"ID":"26c98e8f-b78d-48f5-967b-de9f3cf84e6a","Type":"ContainerDied","Data":"e5f9d5a08849014d565612bc13a01a69e0f37ee43ce820a89797f3f1ee5709fe"}
Jul 17 01:11:53.550926 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:53.550520    2573 scope.go:117] "RemoveContainer" containerID="6c32368060ee110222b3d25efd4ef741dcb91d46e3fea31d7f8d551cc53b3298"
Jul 17 01:11:53.550926 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:53.550815    2573 scope.go:117] "RemoveContainer" containerID="e5f9d5a08849014d565612bc13a01a69e0f37ee43ce820a89797f3f1ee5709fe"
Jul 17 01:11:53.551076 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:11:53.551043    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-6h9vb_odh-ai-gateway-infra(26c98e8f-b78d-48f5-967b-de9f3cf84e6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a"
Jul 17 01:11:53.573518 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:53.573490    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"]
Jul 17 01:11:54.697679 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:54.697655    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"
Jul 17 01:11:54.769859 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:54.769832    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c98e8f-b78d-48f5-967b-de9f3cf84e6a-maas-api-service-ca\") pod \"26c98e8f-b78d-48f5-967b-de9f3cf84e6a\" (UID: \"26c98e8f-b78d-48f5-967b-de9f3cf84e6a\") "
Jul 17 01:11:54.770151 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:54.770129    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c98e8f-b78d-48f5-967b-de9f3cf84e6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26c98e8f-b78d-48f5-967b-de9f3cf84e6a" (UID: "26c98e8f-b78d-48f5-967b-de9f3cf84e6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:11:54.870522 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:54.870461    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c98e8f-b78d-48f5-967b-de9f3cf84e6a-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:11:55.565124 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:55.565088    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"
Jul 17 01:11:55.565124 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:55.565102    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb" event={"ID":"26c98e8f-b78d-48f5-967b-de9f3cf84e6a","Type":"ContainerDied","Data":"5d0ba549566e9e10948ca3a40727dc17ae4e0ed1fce106bb9e9d80552820f647"}
Jul 17 01:11:55.565367 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:55.565154    2573 scope.go:117] "RemoveContainer" containerID="e5f9d5a08849014d565612bc13a01a69e0f37ee43ce820a89797f3f1ee5709fe"
Jul 17 01:11:55.587067 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:55.587041    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"]
Jul 17 01:11:55.590678 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:55.590659    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-6h9vb"]
Jul 17 01:11:55.968595 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:11:55.968526    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" path="/var/lib/kubelet/pods/26c98e8f-b78d-48f5-967b-de9f3cf84e6a/volumes"
Jul 17 01:12:06.975416 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:06.975381    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"]
Jul 17 01:12:06.976035 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:06.975710    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq" podUID="09d3b57a-e993-43b7-ae29-da8a4352e731" containerName="maas-api" containerID="cri-o://cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3" gracePeriod=30
Jul 17 01:12:07.423335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.423312    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:12:07.578054 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.577990    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d3b57a-e993-43b7-ae29-da8a4352e731-maas-api-tls\") pod \"09d3b57a-e993-43b7-ae29-da8a4352e731\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") "
Jul 17 01:12:07.578054 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.578028    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdvx9\" (UniqueName: \"kubernetes.io/projected/09d3b57a-e993-43b7-ae29-da8a4352e731-kube-api-access-pdvx9\") pod \"09d3b57a-e993-43b7-ae29-da8a4352e731\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") "
Jul 17 01:12:07.578235 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.578072    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d3b57a-e993-43b7-ae29-da8a4352e731-openshift-service-ca\") pod \"09d3b57a-e993-43b7-ae29-da8a4352e731\" (UID: \"09d3b57a-e993-43b7-ae29-da8a4352e731\") "
Jul 17 01:12:07.578489 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.578459    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d3b57a-e993-43b7-ae29-da8a4352e731-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "09d3b57a-e993-43b7-ae29-da8a4352e731" (UID: "09d3b57a-e993-43b7-ae29-da8a4352e731"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:12:07.580026 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.580000    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d3b57a-e993-43b7-ae29-da8a4352e731-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "09d3b57a-e993-43b7-ae29-da8a4352e731" (UID: "09d3b57a-e993-43b7-ae29-da8a4352e731"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 01:12:07.580161 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.580126    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d3b57a-e993-43b7-ae29-da8a4352e731-kube-api-access-pdvx9" (OuterVolumeSpecName: "kube-api-access-pdvx9") pod "09d3b57a-e993-43b7-ae29-da8a4352e731" (UID: "09d3b57a-e993-43b7-ae29-da8a4352e731"). InnerVolumeSpecName "kube-api-access-pdvx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:12:07.632975 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.632949    2573 generic.go:356] "Generic (PLEG): container finished" podID="09d3b57a-e993-43b7-ae29-da8a4352e731" containerID="cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3" exitCode=0
Jul 17 01:12:07.633093 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.632992    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"
Jul 17 01:12:07.633093 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.633043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq" event={"ID":"09d3b57a-e993-43b7-ae29-da8a4352e731","Type":"ContainerDied","Data":"cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3"}
Jul 17 01:12:07.633226 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.633109    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq" event={"ID":"09d3b57a-e993-43b7-ae29-da8a4352e731","Type":"ContainerDied","Data":"92774de6675a78a0225f38dfecba9807c624b2ac8ece0d86e82573f7275f0e1f"}
Jul 17 01:12:07.633226 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.633140    2573 scope.go:117] "RemoveContainer" containerID="cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3"
Jul 17 01:12:07.643502 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.643472    2573 scope.go:117] "RemoveContainer" containerID="cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3"
Jul 17 01:12:07.643839 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:07.643818    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3\": container with ID starting with cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3 not found: ID does not exist" containerID="cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3"
Jul 17 01:12:07.643927 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.643851    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3"} err="failed to get container status \"cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3\": rpc error: code = NotFound desc = could not find container \"cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3\": container with ID starting with cb358414cd850e443706fe6d2ec5ea0964863caa65a59395a44ce570ead85ba3 not found: ID does not exist"
Jul 17 01:12:07.653651 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.653631    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"]
Jul 17 01:12:07.656909 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.656889    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e1dd24-688c878955-2ftlq"]
Jul 17 01:12:07.678584 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.678567    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdvx9\" (UniqueName: \"kubernetes.io/projected/09d3b57a-e993-43b7-ae29-da8a4352e731-kube-api-access-pdvx9\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:07.678670 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.678586    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d3b57a-e993-43b7-ae29-da8a4352e731-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:07.678670 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.678600    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d3b57a-e993-43b7-ae29-da8a4352e731-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:07.968875 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:07.968769    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d3b57a-e993-43b7-ae29-da8a4352e731" path="/var/lib/kubelet/pods/09d3b57a-e993-43b7-ae29-da8a4352e731/volumes"
Jul 17 01:12:22.545564 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545521    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"]
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545924    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545938    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545947    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545953    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545961    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d3b57a-e993-43b7-ae29-da8a4352e731" containerName="maas-api"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545967    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d3b57a-e993-43b7-ae29-da8a4352e731" containerName="maas-api"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545980    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e170e529-2011-49a7-870b-26e893e82b39" containerName="test-curl-7469-139661443547760"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.545986    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e170e529-2011-49a7-870b-26e893e82b39" containerName="test-curl-7469-139661443547760"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.546047    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d3b57a-e993-43b7-ae29-da8a4352e731" containerName="maas-api"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.546055    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e170e529-2011-49a7-870b-26e893e82b39" containerName="test-curl-7469-139661443547760"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.546061    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.546067    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:22.546077 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.546073    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:22.551282 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.551260    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"
Jul 17 01:12:22.553217 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.553198    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:12:22.557219 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.557193    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"]
Jul 17 01:12:22.698620 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.698587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f07b670-fcbb-40bb-b7df-d0ed4f78d588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz\" (UID: \"3f07b670-fcbb-40bb-b7df-d0ed4f78d588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"
Jul 17 01:12:22.799427 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.799360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f07b670-fcbb-40bb-b7df-d0ed4f78d588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz\" (UID: \"3f07b670-fcbb-40bb-b7df-d0ed4f78d588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"
Jul 17 01:12:22.799995 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.799975    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f07b670-fcbb-40bb-b7df-d0ed4f78d588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz\" (UID: \"3f07b670-fcbb-40bb-b7df-d0ed4f78d588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"
Jul 17 01:12:22.863621 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.863594    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"
Jul 17 01:12:22.999362 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:22.999335    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"]
Jul 17 01:12:23.000991 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:12:23.000960    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f07b670_fcbb_40bb_b7df_d0ed4f78d588.slice/crio-d650fce19351b25f109e9ffcf0a29ebd2b209910f57718c3ea8c78641679ad4e WatchSource:0}: Error finding container d650fce19351b25f109e9ffcf0a29ebd2b209910f57718c3ea8c78641679ad4e: Status 404 returned error can't find the container with id d650fce19351b25f109e9ffcf0a29ebd2b209910f57718c3ea8c78641679ad4e
Jul 17 01:12:23.724577 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:23.724530    2573 generic.go:356] "Generic (PLEG): container finished" podID="3f07b670-fcbb-40bb-b7df-d0ed4f78d588" containerID="92e3d30da04554bbf577edb3ef5717fad21bbabcee18cadf6851d1a944a3ce89" exitCode=0
Jul 17 01:12:23.725209 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:23.724609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz" event={"ID":"3f07b670-fcbb-40bb-b7df-d0ed4f78d588","Type":"ContainerDied","Data":"92e3d30da04554bbf577edb3ef5717fad21bbabcee18cadf6851d1a944a3ce89"}
Jul 17 01:12:23.725209 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:23.724671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz" event={"ID":"3f07b670-fcbb-40bb-b7df-d0ed4f78d588","Type":"ContainerStarted","Data":"d650fce19351b25f109e9ffcf0a29ebd2b209910f57718c3ea8c78641679ad4e"}
Jul 17 01:12:24.867433 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:24.867410    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"
Jul 17 01:12:25.016090 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:25.016058    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f07b670-fcbb-40bb-b7df-d0ed4f78d588-maas-api-service-ca\") pod \"3f07b670-fcbb-40bb-b7df-d0ed4f78d588\" (UID: \"3f07b670-fcbb-40bb-b7df-d0ed4f78d588\") "
Jul 17 01:12:25.016398 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:25.016376    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f07b670-fcbb-40bb-b7df-d0ed4f78d588-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f07b670-fcbb-40bb-b7df-d0ed4f78d588" (UID: "3f07b670-fcbb-40bb-b7df-d0ed4f78d588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:12:25.117217 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:25.117180    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f07b670-fcbb-40bb-b7df-d0ed4f78d588-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:25.736265 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:25.736239    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz"
Jul 17 01:12:25.736468 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:25.736236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz" event={"ID":"3f07b670-fcbb-40bb-b7df-d0ed4f78d588","Type":"ContainerDied","Data":"d650fce19351b25f109e9ffcf0a29ebd2b209910f57718c3ea8c78641679ad4e"}
Jul 17 01:12:25.736468 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:25.736358    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d650fce19351b25f109e9ffcf0a29ebd2b209910f57718c3ea8c78641679ad4e"
Jul 17 01:12:26.772785 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.772753    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"]
Jul 17 01:12:26.773161 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.773145    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:26.773161 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.773157    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c98e8f-b78d-48f5-967b-de9f3cf84e6a" containerName="revoke-keys"
Jul 17 01:12:26.773240 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.773178    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f07b670-fcbb-40bb-b7df-d0ed4f78d588" containerName="revoke-keys"
Jul 17 01:12:26.773240 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.773184    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f07b670-fcbb-40bb-b7df-d0ed4f78d588" containerName="revoke-keys"
Jul 17 01:12:26.773240 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.773237    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f07b670-fcbb-40bb-b7df-d0ed4f78d588" containerName="revoke-keys"
Jul 17 01:12:26.776476 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.776460    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"
Jul 17 01:12:26.778487 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.778458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:12:26.781776 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.781744    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"]
Jul 17 01:12:26.931455 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:26.931428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d1b0f-694c-4473-90df-db0760370b14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-d89m6\" (UID: \"c26d1b0f-694c-4473-90df-db0760370b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"
Jul 17 01:12:27.031968 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:27.031892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d1b0f-694c-4473-90df-db0760370b14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-d89m6\" (UID: \"c26d1b0f-694c-4473-90df-db0760370b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"
Jul 17 01:12:27.032459 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:27.032440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d1b0f-694c-4473-90df-db0760370b14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-d89m6\" (UID: \"c26d1b0f-694c-4473-90df-db0760370b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"
Jul 17 01:12:27.089188 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:27.089165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"
Jul 17 01:12:27.218123 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:27.218091    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"]
Jul 17 01:12:27.218857 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:12:27.218828    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26d1b0f_694c_4473_90df_db0760370b14.slice/crio-e25702f04c9b7c66d5d4efdc539eeafdda69f1d01f6c81461e48adb843e1f13a WatchSource:0}: Error finding container e25702f04c9b7c66d5d4efdc539eeafdda69f1d01f6c81461e48adb843e1f13a: Status 404 returned error can't find the container with id e25702f04c9b7c66d5d4efdc539eeafdda69f1d01f6c81461e48adb843e1f13a
Jul 17 01:12:27.751526 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:27.751441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" event={"ID":"c26d1b0f-694c-4473-90df-db0760370b14","Type":"ContainerStarted","Data":"4ff70f973d9c4a19a66ebb11c429a767e293883850e973a2d2310feb6d439c76"}
Jul 17 01:12:27.751526 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:27.751499    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" event={"ID":"c26d1b0f-694c-4473-90df-db0760370b14","Type":"ContainerStarted","Data":"e25702f04c9b7c66d5d4efdc539eeafdda69f1d01f6c81461e48adb843e1f13a"}
Jul 17 01:12:27.768417 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:27.768375    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" podStartSLOduration=1.768361852 podStartE2EDuration="1.768361852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:12:27.766807883 +0000 UTC m=+3342.342960447" watchObservedRunningTime="2026-07-17 01:12:27.768361852 +0000 UTC m=+3342.344514440"
Jul 17 01:12:28.757946 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:28.757910    2573 generic.go:356] "Generic (PLEG): container finished" podID="c26d1b0f-694c-4473-90df-db0760370b14" containerID="4ff70f973d9c4a19a66ebb11c429a767e293883850e973a2d2310feb6d439c76" exitCode=6
Jul 17 01:12:28.758383 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:28.757993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" event={"ID":"c26d1b0f-694c-4473-90df-db0760370b14","Type":"ContainerDied","Data":"4ff70f973d9c4a19a66ebb11c429a767e293883850e973a2d2310feb6d439c76"}
Jul 17 01:12:28.758383 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:28.758157    2573 scope.go:117] "RemoveContainer" containerID="4ff70f973d9c4a19a66ebb11c429a767e293883850e973a2d2310feb6d439c76"
Jul 17 01:12:29.765711 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:29.765681    2573 generic.go:356] "Generic (PLEG): container finished" podID="c26d1b0f-694c-4473-90df-db0760370b14" containerID="50b8ebe41b5ef8b04329908eb087d568e2e2cbd0f3f67aae570b1a7477333074" exitCode=6
Jul 17 01:12:29.766125 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:29.765762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" event={"ID":"c26d1b0f-694c-4473-90df-db0760370b14","Type":"ContainerDied","Data":"50b8ebe41b5ef8b04329908eb087d568e2e2cbd0f3f67aae570b1a7477333074"}
Jul 17 01:12:29.766125 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:29.765833    2573 scope.go:117] "RemoveContainer" containerID="4ff70f973d9c4a19a66ebb11c429a767e293883850e973a2d2310feb6d439c76"
Jul 17 01:12:29.766125 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:29.765965    2573 scope.go:117] "RemoveContainer" containerID="50b8ebe41b5ef8b04329908eb087d568e2e2cbd0f3f67aae570b1a7477333074"
Jul 17 01:12:29.766254 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:29.766172    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-d89m6_odh-ai-gateway-infra(c26d1b0f-694c-4473-90df-db0760370b14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" podUID="c26d1b0f-694c-4473-90df-db0760370b14"
Jul 17 01:12:30.773142 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:30.773116    2573 scope.go:117] "RemoveContainer" containerID="50b8ebe41b5ef8b04329908eb087d568e2e2cbd0f3f67aae570b1a7477333074"
Jul 17 01:12:30.773522 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:30.773314    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-d89m6_odh-ai-gateway-infra(c26d1b0f-694c-4473-90df-db0760370b14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" podUID="c26d1b0f-694c-4473-90df-db0760370b14"
Jul 17 01:12:32.950406 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:32.950146    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"]
Jul 17 01:12:32.950406 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:32.950378    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" podUID="9dbabf49-66cb-4b79-8d3d-05d226d7c22e" containerName="maas-api" containerID="cri-o://3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd" gracePeriod=30
Jul 17 01:12:33.037418 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.037378    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" podUID="9dbabf49-66cb-4b79-8d3d-05d226d7c22e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": dial tcp 10.133.0.161:8443: connect: connection refused"
Jul 17 01:12:33.397502 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.397480    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:12:33.483888 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.483855    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-openshift-service-ca\") pod \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") "
Jul 17 01:12:33.484030 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.483917    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbjz2\" (UniqueName: \"kubernetes.io/projected/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-kube-api-access-zbjz2\") pod \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") "
Jul 17 01:12:33.484216 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.484198    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9dbabf49-66cb-4b79-8d3d-05d226d7c22e" (UID: "9dbabf49-66cb-4b79-8d3d-05d226d7c22e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:12:33.485955 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.485903    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-kube-api-access-zbjz2" (OuterVolumeSpecName: "kube-api-access-zbjz2") pod "9dbabf49-66cb-4b79-8d3d-05d226d7c22e" (UID: "9dbabf49-66cb-4b79-8d3d-05d226d7c22e"). InnerVolumeSpecName "kube-api-access-zbjz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 01:12:33.584547 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.584522    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls\") pod \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\" (UID: \"9dbabf49-66cb-4b79-8d3d-05d226d7c22e\") "
Jul 17 01:12:33.584672 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.584649    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbjz2\" (UniqueName: \"kubernetes.io/projected/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-kube-api-access-zbjz2\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:33.584672 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.584660    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-openshift-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:33.586410 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.586388    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9dbabf49-66cb-4b79-8d3d-05d226d7c22e" (UID: "9dbabf49-66cb-4b79-8d3d-05d226d7c22e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 01:12:33.684948 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.684924    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbabf49-66cb-4b79-8d3d-05d226d7c22e-maas-api-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:33.791474 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.791445    2573 generic.go:356] "Generic (PLEG): container finished" podID="9dbabf49-66cb-4b79-8d3d-05d226d7c22e" containerID="3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd" exitCode=0
Jul 17 01:12:33.791581 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.791488    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"
Jul 17 01:12:33.791581 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.791534    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" event={"ID":"9dbabf49-66cb-4b79-8d3d-05d226d7c22e","Type":"ContainerDied","Data":"3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd"}
Jul 17 01:12:33.791701 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.791590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b" event={"ID":"9dbabf49-66cb-4b79-8d3d-05d226d7c22e","Type":"ContainerDied","Data":"e42b529db054261aa7754c4e7f3ff8858ba92e132fce5c5270926a0524698849"}
Jul 17 01:12:33.791701 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.791622    2573 scope.go:117] "RemoveContainer" containerID="3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd"
Jul 17 01:12:33.800383 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.800369    2573 scope.go:117] "RemoveContainer" containerID="3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd"
Jul 17 01:12:33.800616 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:33.800598    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd\": container with ID starting with 3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd not found: ID does not exist" containerID="3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd"
Jul 17 01:12:33.800718 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.800623    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd"} err="failed to get container status \"3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd\": rpc error: code = NotFound desc = could not find container \"3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd\": container with ID starting with 3de0ae7cae6c2693935569261d45bb54cda92cc7c4675ff5091847bb9ea1b1cd not found: ID does not exist"
Jul 17 01:12:33.812163 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.812141    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"]
Jul 17 01:12:33.815819 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.815800    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ac54e2-78cff6d7db-k8w4b"]
Jul 17 01:12:33.967892 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:33.967861    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dbabf49-66cb-4b79-8d3d-05d226d7c22e" path="/var/lib/kubelet/pods/9dbabf49-66cb-4b79-8d3d-05d226d7c22e/volumes"
Jul 17 01:12:43.962253 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:43.962227    2573 scope.go:117] "RemoveContainer" containerID="50b8ebe41b5ef8b04329908eb087d568e2e2cbd0f3f67aae570b1a7477333074"
Jul 17 01:12:44.856705 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:44.856634    2573 generic.go:356] "Generic (PLEG): container finished" podID="c26d1b0f-694c-4473-90df-db0760370b14" containerID="bff84fc2dd556426c75dcfdf9aa9dfe7a2e6592a8231136101c525aed69ea9ff" exitCode=6
Jul 17 01:12:44.856846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:44.856712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" event={"ID":"c26d1b0f-694c-4473-90df-db0760370b14","Type":"ContainerDied","Data":"bff84fc2dd556426c75dcfdf9aa9dfe7a2e6592a8231136101c525aed69ea9ff"}
Jul 17 01:12:44.856846 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:44.856772    2573 scope.go:117] "RemoveContainer" containerID="50b8ebe41b5ef8b04329908eb087d568e2e2cbd0f3f67aae570b1a7477333074"
Jul 17 01:12:44.857058 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:44.857040    2573 scope.go:117] "RemoveContainer" containerID="bff84fc2dd556426c75dcfdf9aa9dfe7a2e6592a8231136101c525aed69ea9ff"
Jul 17 01:12:44.857249 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:44.857229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-d89m6_odh-ai-gateway-infra(c26d1b0f-694c-4473-90df-db0760370b14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" podUID="c26d1b0f-694c-4473-90df-db0760370b14"
Jul 17 01:12:45.880429 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:45.880399    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"]
Jul 17 01:12:46.013767 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.013739    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"
Jul 17 01:12:46.077720 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.077693    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d1b0f-694c-4473-90df-db0760370b14-maas-api-service-ca\") pod \"c26d1b0f-694c-4473-90df-db0760370b14\" (UID: \"c26d1b0f-694c-4473-90df-db0760370b14\") "
Jul 17 01:12:46.078059 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.078035    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26d1b0f-694c-4473-90df-db0760370b14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c26d1b0f-694c-4473-90df-db0760370b14" (UID: "c26d1b0f-694c-4473-90df-db0760370b14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:12:46.178944 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.178891    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d1b0f-694c-4473-90df-db0760370b14-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:12:46.871176 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.871135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6" event={"ID":"c26d1b0f-694c-4473-90df-db0760370b14","Type":"ContainerDied","Data":"e25702f04c9b7c66d5d4efdc539eeafdda69f1d01f6c81461e48adb843e1f13a"}
Jul 17 01:12:46.871370 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.871201    2573 scope.go:117] "RemoveContainer" containerID="bff84fc2dd556426c75dcfdf9aa9dfe7a2e6592a8231136101c525aed69ea9ff"
Jul 17 01:12:46.871370 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.871150    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"
Jul 17 01:12:46.891907 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.891886    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"]
Jul 17 01:12:46.895387 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:46.895367    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-d89m6"]
Jul 17 01:12:47.969115 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:47.969083    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26d1b0f-694c-4473-90df-db0760370b14" path="/var/lib/kubelet/pods/c26d1b0f-694c-4473-90df-db0760370b14/volumes"
Jul 17 01:12:49.494071 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494039    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"]
Jul 17 01:12:49.494425 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494366    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:49.494425 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494376    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:49.494425 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494395    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dbabf49-66cb-4b79-8d3d-05d226d7c22e" containerName="maas-api"
Jul 17 01:12:49.494425 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494401    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dbabf49-66cb-4b79-8d3d-05d226d7c22e" containerName="maas-api"
Jul 17 01:12:49.494425 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494412    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:49.494425 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494418    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:49.494643 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494469    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:49.494643 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494476    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:49.494643 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.494484    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dbabf49-66cb-4b79-8d3d-05d226d7c22e" containerName="maas-api"
Jul 17 01:12:49.499078 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.499061    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"
Jul 17 01:12:49.501064 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.501028    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:12:49.504143 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.504115    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"]
Jul 17 01:12:49.603695 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.603666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d809380-837e-471e-a480-029ec0aa80c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-srzxz\" (UID: \"2d809380-837e-471e-a480-029ec0aa80c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"
Jul 17 01:12:49.704941 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.704914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d809380-837e-471e-a480-029ec0aa80c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-srzxz\" (UID: \"2d809380-837e-471e-a480-029ec0aa80c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"
Jul 17 01:12:49.705448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.705432    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d809380-837e-471e-a480-029ec0aa80c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-srzxz\" (UID: \"2d809380-837e-471e-a480-029ec0aa80c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"
Jul 17 01:12:49.811129 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.811105    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"
Jul 17 01:12:49.941147 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:49.941070    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"]
Jul 17 01:12:49.942455 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:12:49.942424    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d809380_837e_471e_a480_029ec0aa80c1.slice/crio-371ce1f2ec69379051902c2cd78d2d1959fabce663df551c970be12b63a6882a WatchSource:0}: Error finding container 371ce1f2ec69379051902c2cd78d2d1959fabce663df551c970be12b63a6882a: Status 404 returned error can't find the container with id 371ce1f2ec69379051902c2cd78d2d1959fabce663df551c970be12b63a6882a
Jul 17 01:12:50.897307 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:50.897227    2573 generic.go:356] "Generic (PLEG): container finished" podID="2d809380-837e-471e-a480-029ec0aa80c1" containerID="e9b01480d0801afdeabbbd925a3626bd76377e779f04eb71a959f23ab2a804b8" exitCode=6
Jul 17 01:12:50.897736 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:50.897311    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" event={"ID":"2d809380-837e-471e-a480-029ec0aa80c1","Type":"ContainerDied","Data":"e9b01480d0801afdeabbbd925a3626bd76377e779f04eb71a959f23ab2a804b8"}
Jul 17 01:12:50.897736 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:50.897369    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" event={"ID":"2d809380-837e-471e-a480-029ec0aa80c1","Type":"ContainerStarted","Data":"371ce1f2ec69379051902c2cd78d2d1959fabce663df551c970be12b63a6882a"}
Jul 17 01:12:50.897736 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:50.897444    2573 scope.go:117] "RemoveContainer" containerID="e9b01480d0801afdeabbbd925a3626bd76377e779f04eb71a959f23ab2a804b8"
Jul 17 01:12:51.350648 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.350614    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"]
Jul 17 01:12:51.351014 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.351002    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:51.351069 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.351016    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:51.351108 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.351072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26d1b0f-694c-4473-90df-db0760370b14" containerName="revoke-keys"
Jul 17 01:12:51.354102 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.354085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"
Jul 17 01:12:51.358965 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.358943    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"]
Jul 17 01:12:51.419728 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.419663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5719043e-4fff-4830-be1a-af4c9b34711b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh\" (UID: \"5719043e-4fff-4830-be1a-af4c9b34711b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"
Jul 17 01:12:51.520963 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.520937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5719043e-4fff-4830-be1a-af4c9b34711b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh\" (UID: \"5719043e-4fff-4830-be1a-af4c9b34711b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"
Jul 17 01:12:51.521486 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.521469    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5719043e-4fff-4830-be1a-af4c9b34711b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh\" (UID: \"5719043e-4fff-4830-be1a-af4c9b34711b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"
Jul 17 01:12:51.666103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.666075    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"
Jul 17 01:12:51.905598 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.905567    2573 generic.go:356] "Generic (PLEG): container finished" podID="2d809380-837e-471e-a480-029ec0aa80c1" containerID="00b96269e5eeb394afa033eabcefca556580c267f71dfb13e3b9a3c9e4129eb2" exitCode=6
Jul 17 01:12:51.906027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.905660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" event={"ID":"2d809380-837e-471e-a480-029ec0aa80c1","Type":"ContainerDied","Data":"00b96269e5eeb394afa033eabcefca556580c267f71dfb13e3b9a3c9e4129eb2"}
Jul 17 01:12:51.906027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.905716    2573 scope.go:117] "RemoveContainer" containerID="e9b01480d0801afdeabbbd925a3626bd76377e779f04eb71a959f23ab2a804b8"
Jul 17 01:12:51.906027 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:51.905859    2573 scope.go:117] "RemoveContainer" containerID="00b96269e5eeb394afa033eabcefca556580c267f71dfb13e3b9a3c9e4129eb2"
Jul 17 01:12:51.906215 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:51.906086    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-srzxz_odh-ai-gateway-infra(2d809380-837e-471e-a480-029ec0aa80c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" podUID="2d809380-837e-471e-a480-029ec0aa80c1"
Jul 17 01:12:52.000525 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:52.000502    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"]
Jul 17 01:12:52.002234 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:12:52.002211    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5719043e_4fff_4830_be1a_af4c9b34711b.slice/crio-e57e2b4ea8f2bd24c1b26591a0394986b6819cdcd7818333000b82f4ec7c030b WatchSource:0}: Error finding container e57e2b4ea8f2bd24c1b26591a0394986b6819cdcd7818333000b82f4ec7c030b: Status 404 returned error can't find the container with id e57e2b4ea8f2bd24c1b26591a0394986b6819cdcd7818333000b82f4ec7c030b
Jul 17 01:12:52.912971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:52.912895    2573 generic.go:356] "Generic (PLEG): container finished" podID="5719043e-4fff-4830-be1a-af4c9b34711b" containerID="0d841a1ec52fd2789f069b4963f24429af9571ea45c926f9841caf316fa65839" exitCode=6
Jul 17 01:12:52.912971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:52.912940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" event={"ID":"5719043e-4fff-4830-be1a-af4c9b34711b","Type":"ContainerDied","Data":"0d841a1ec52fd2789f069b4963f24429af9571ea45c926f9841caf316fa65839"}
Jul 17 01:12:52.913392 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:52.913036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" event={"ID":"5719043e-4fff-4830-be1a-af4c9b34711b","Type":"ContainerStarted","Data":"e57e2b4ea8f2bd24c1b26591a0394986b6819cdcd7818333000b82f4ec7c030b"}
Jul 17 01:12:52.913392 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:52.913216    2573 scope.go:117] "RemoveContainer" containerID="0d841a1ec52fd2789f069b4963f24429af9571ea45c926f9841caf316fa65839"
Jul 17 01:12:52.914934 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:52.914917    2573 scope.go:117] "RemoveContainer" containerID="00b96269e5eeb394afa033eabcefca556580c267f71dfb13e3b9a3c9e4129eb2"
Jul 17 01:12:52.915117 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:52.915094    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-srzxz_odh-ai-gateway-infra(2d809380-837e-471e-a480-029ec0aa80c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" podUID="2d809380-837e-471e-a480-029ec0aa80c1"
Jul 17 01:12:53.767744 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.767714    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"]
Jul 17 01:12:53.771387 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.771367    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"
Jul 17 01:12:53.777480 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.777419    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"]
Jul 17 01:12:53.841176 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.841109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85714b14-cc44-4b5f-aa80-14402fb49423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt\" (UID: \"85714b14-cc44-4b5f-aa80-14402fb49423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"
Jul 17 01:12:53.922418 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.922389    2573 generic.go:356] "Generic (PLEG): container finished" podID="5719043e-4fff-4830-be1a-af4c9b34711b" containerID="09eda01fe729eb1447e1c55a3eb090335843d8d761f357d9f3274f0bb4213e7c" exitCode=6
Jul 17 01:12:53.922783 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.922461    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" event={"ID":"5719043e-4fff-4830-be1a-af4c9b34711b","Type":"ContainerDied","Data":"09eda01fe729eb1447e1c55a3eb090335843d8d761f357d9f3274f0bb4213e7c"}
Jul 17 01:12:53.922783 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.922523    2573 scope.go:117] "RemoveContainer" containerID="0d841a1ec52fd2789f069b4963f24429af9571ea45c926f9841caf316fa65839"
Jul 17 01:12:53.922783 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.922717    2573 scope.go:117] "RemoveContainer" containerID="09eda01fe729eb1447e1c55a3eb090335843d8d761f357d9f3274f0bb4213e7c"
Jul 17 01:12:53.922987 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:53.922964    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh_odh-ai-gateway-infra(5719043e-4fff-4830-be1a-af4c9b34711b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" podUID="5719043e-4fff-4830-be1a-af4c9b34711b"
Jul 17 01:12:53.941546 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.941520    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85714b14-cc44-4b5f-aa80-14402fb49423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt\" (UID: \"85714b14-cc44-4b5f-aa80-14402fb49423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"
Jul 17 01:12:53.942112 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:53.942096    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85714b14-cc44-4b5f-aa80-14402fb49423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt\" (UID: \"85714b14-cc44-4b5f-aa80-14402fb49423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"
Jul 17 01:12:54.084140 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.084109    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"
Jul 17 01:12:54.215918 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.215889    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"]
Jul 17 01:12:54.216653 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:12:54.216631    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85714b14_cc44_4b5f_aa80_14402fb49423.slice/crio-5deef50b7fb439d55209705777dc8230f23542a0b6215c1eddd11b5cbee29d7d WatchSource:0}: Error finding container 5deef50b7fb439d55209705777dc8230f23542a0b6215c1eddd11b5cbee29d7d: Status 404 returned error can't find the container with id 5deef50b7fb439d55209705777dc8230f23542a0b6215c1eddd11b5cbee29d7d
Jul 17 01:12:54.325921 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.325881    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-965b4997-zn7mh_a21cd433-0636-49ed-871e-67a383ac3c87/manager/0.log"
Jul 17 01:12:54.432848 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.432760    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xvwgt_7a35fd16-bf16-46a5-a26a-8e3d83d0b66b/server/0.log"
Jul 17 01:12:54.551831 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.551801    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v92p8_8b2d5d3c-64f4-4d5c-a629-60819a1262a6/manager/0.log"
Jul 17 01:12:54.655343 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.655320    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-8mc6b_47849ce6-96d2-4b16-9ce6-9b6765a6b5ed/copy-manifests/0.log"
Jul 17 01:12:54.672776 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.672746    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-8mc6b_47849ce6-96d2-4b16-9ce6-9b6765a6b5ed/manager/0.log"
Jul 17 01:12:54.930116 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.930088    2573 generic.go:356] "Generic (PLEG): container finished" podID="85714b14-cc44-4b5f-aa80-14402fb49423" containerID="d1b7cefacb39b5a26aaca945602608e1e6f305500a13375612d7d40873d7fd40" exitCode=6
Jul 17 01:12:54.930520 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.930175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" event={"ID":"85714b14-cc44-4b5f-aa80-14402fb49423","Type":"ContainerDied","Data":"d1b7cefacb39b5a26aaca945602608e1e6f305500a13375612d7d40873d7fd40"}
Jul 17 01:12:54.930520 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.930226    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" event={"ID":"85714b14-cc44-4b5f-aa80-14402fb49423","Type":"ContainerStarted","Data":"5deef50b7fb439d55209705777dc8230f23542a0b6215c1eddd11b5cbee29d7d"}
Jul 17 01:12:54.930520 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.930367    2573 scope.go:117] "RemoveContainer" containerID="d1b7cefacb39b5a26aaca945602608e1e6f305500a13375612d7d40873d7fd40"
Jul 17 01:12:54.931872 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:54.931858    2573 scope.go:117] "RemoveContainer" containerID="09eda01fe729eb1447e1c55a3eb090335843d8d761f357d9f3274f0bb4213e7c"
Jul 17 01:12:54.932018 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:54.932003    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh_odh-ai-gateway-infra(5719043e-4fff-4830-be1a-af4c9b34711b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" podUID="5719043e-4fff-4830-be1a-af4c9b34711b"
Jul 17 01:12:55.362333 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.362309    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737470-cq2gl_982d352b-b348-4ef2-b927-7bdeb7f4c3a6/cleanup/0.log"
Jul 17 01:12:55.467809 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.467722    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737485-gw87d_9d7dcbf4-f9c7-46b9-8cb1-4583a5a0fae4/cleanup/0.log"
Jul 17 01:12:55.575603 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.575578    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737500-z8wll_2bbd9b93-b24f-4499-888c-d96665aed403/cleanup/0.log"
Jul 17 01:12:55.681614 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.681588    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-4692a03f-srzxz_2d809380-837e-471e-a480-029ec0aa80c1/revoke-keys/1.log"
Jul 17 01:12:55.788677 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.788653    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt_85714b14-cc44-4b5f-aa80-14402fb49423/revoke-keys/0.log"
Jul 17 01:12:55.939405 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.939384    2573 generic.go:356] "Generic (PLEG): container finished" podID="85714b14-cc44-4b5f-aa80-14402fb49423" containerID="5c4ee6333e112c87f59b4fa8e06e77b82fb0ff811cae65772e96f559bb0d5a15" exitCode=6
Jul 17 01:12:55.939761 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.939471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" event={"ID":"85714b14-cc44-4b5f-aa80-14402fb49423","Type":"ContainerDied","Data":"5c4ee6333e112c87f59b4fa8e06e77b82fb0ff811cae65772e96f559bb0d5a15"}
Jul 17 01:12:55.939761 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.939533    2573 scope.go:117] "RemoveContainer" containerID="d1b7cefacb39b5a26aaca945602608e1e6f305500a13375612d7d40873d7fd40"
Jul 17 01:12:55.939761 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.939715    2573 scope.go:117] "RemoveContainer" containerID="5c4ee6333e112c87f59b4fa8e06e77b82fb0ff811cae65772e96f559bb0d5a15"
Jul 17 01:12:55.939940 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:55.939920    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt_odh-ai-gateway-infra(85714b14-cc44-4b5f-aa80-14402fb49423)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" podUID="85714b14-cc44-4b5f-aa80-14402fb49423"
Jul 17 01:12:55.995149 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:55.995128    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-49fd04-j9pz4_115ab490-eada-4fd1-9214-b204629c67d3/revoke-keys/0.log"
Jul 17 01:12:56.100592 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.100548    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-27f370-m56kr_e5bcf850-8ef9-4069-8b1e-8f391513b3b4/revoke-keys/0.log"
Jul 17 01:12:56.204586 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.204564    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-e1e207-q4jc2_2856d5b3-ccc6-4247-b9ef-75078e6c3c1d/revoke-keys/0.log"
Jul 17 01:12:56.307720 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.307694    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh_5719043e-4fff-4830-be1a-af4c9b34711b/revoke-keys/1.log"
Jul 17 01:12:56.409334 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.409276    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-0a075aab-t6jql_dd329768-544c-4e2f-8ce5-b9bb9d00a0e7/revoke-keys/0.log"
Jul 17 01:12:56.512522 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.512494    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-380fc75a-8prlc_0ba3fad2-2590-453a-9713-392df9302bdf/revoke-keys/0.log"
Jul 17 01:12:56.716852 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.716760    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-ac54e2-8bsgz_3f07b670-fcbb-40bb-b7df-d0ed4f78d588/revoke-keys/0.log"
Jul 17 01:12:56.820840 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.820815    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9jbtn_11c30936-0258-4e90-8759-e8e550709078/postgres/0.log"
Jul 17 01:12:56.947709 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:56.947687    2573 scope.go:117] "RemoveContainer" containerID="5c4ee6333e112c87f59b4fa8e06e77b82fb0ff811cae65772e96f559bb0d5a15"
Jul 17 01:12:56.948122 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:12:56.947903    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt_odh-ai-gateway-infra(85714b14-cc44-4b5f-aa80-14402fb49423)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" podUID="85714b14-cc44-4b5f-aa80-14402fb49423"
Jul 17 01:12:57.549725 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.549692    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc_db088276-f18e-4fd4-ab03-8fb01725451e/util/0.log"
Jul 17 01:12:57.555139 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.555112    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc_db088276-f18e-4fd4-ab03-8fb01725451e/pull/0.log"
Jul 17 01:12:57.560103 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.560080    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc_db088276-f18e-4fd4-ab03-8fb01725451e/extract/0.log"
Jul 17 01:12:57.661666 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.661647    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp_2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41/extract/0.log"
Jul 17 01:12:57.667210 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.667190    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp_2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41/util/0.log"
Jul 17 01:12:57.673010 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.672990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp_2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41/pull/0.log"
Jul 17 01:12:57.783859 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.783834    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk_84c1c8fb-4cef-450e-aa29-69685b2384e3/extract/0.log"
Jul 17 01:12:57.789050 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.789030    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk_84c1c8fb-4cef-450e-aa29-69685b2384e3/util/0.log"
Jul 17 01:12:57.794753 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.794735    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk_84c1c8fb-4cef-450e-aa29-69685b2384e3/pull/0.log"
Jul 17 01:12:57.898180 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.898119    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq_1b84bdb2-14fe-4da0-b015-f5c40e62a2e8/util/0.log"
Jul 17 01:12:57.903684 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.903665    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq_1b84bdb2-14fe-4da0-b015-f5c40e62a2e8/pull/0.log"
Jul 17 01:12:57.909221 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:57.909201    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq_1b84bdb2-14fe-4da0-b015-f5c40e62a2e8/extract/0.log"
Jul 17 01:12:58.133041 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:58.132995    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gr8xz_f7b46a10-a42d-40b5-8ad6-986d6740be80/manager/0.log"
Jul 17 01:12:58.239051 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:58.238991    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rjmbk_750eefa1-5519-4cb9-bebc-58e9d2a55d48/manager/0.log"
Jul 17 01:12:58.348551 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:58.348530    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kszxw_c02072b4-cf4f-4566-8fc0-e5b552507e54/kuadrant-console-plugin/0.log"
Jul 17 01:12:58.463480 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:58.463457    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6mc2k_cbc87a2e-6674-4bb4-ae2e-018e6362a9a6/registry-server/0.log"
Jul 17 01:12:58.582692 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:58.582669    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bjvkl_45577182-8092-4ac7-b438-cbb85b50724a/manager/0.log"
Jul 17 01:12:59.246594 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:59.246564    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jndrl_48e44104-5be3-4c4f-b021-e22713374ecf/discovery/0.log"
Jul 17 01:12:59.354638 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:59.354616    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-75nvg_85e79f05-6be2-4c22-b098-89eb2450e25c/kube-auth-proxy/0.log"
Jul 17 01:12:59.685527 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:12:59.685499    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pm4jz_af6b4198-05cf-4598-a670-096a6f7cf460/payload-pre-processing/0.log"
Jul 17 01:13:00.229991 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:00.229964    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-hms46_cb5914f6-f2c7-4c76-8a4b-6bde1e6741ba/main/0.log"
Jul 17 01:13:00.331975 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:00.331947    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-b6zdk_02c2e277-8128-4838-b7e5-b82baf799fce/main/0.log"
Jul 17 01:13:07.962999 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:07.962967    2573 scope.go:117] "RemoveContainer" containerID="09eda01fe729eb1447e1c55a3eb090335843d8d761f357d9f3274f0bb4213e7c"
Jul 17 01:13:07.963358 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:07.963066    2573 scope.go:117] "RemoveContainer" containerID="00b96269e5eeb394afa033eabcefca556580c267f71dfb13e3b9a3c9e4129eb2"
Jul 17 01:13:08.963101 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:08.963045    2573 scope.go:117] "RemoveContainer" containerID="5c4ee6333e112c87f59b4fa8e06e77b82fb0ff811cae65772e96f559bb0d5a15"
Jul 17 01:13:09.011922 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.011899    2573 generic.go:356] "Generic (PLEG): container finished" podID="5719043e-4fff-4830-be1a-af4c9b34711b" containerID="acea6139554f9f9d0dc62c1886ff9d217af87b5904d6763b82ab38f207602550" exitCode=6
Jul 17 01:13:09.012023 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.011972    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" event={"ID":"5719043e-4fff-4830-be1a-af4c9b34711b","Type":"ContainerDied","Data":"acea6139554f9f9d0dc62c1886ff9d217af87b5904d6763b82ab38f207602550"}
Jul 17 01:13:09.012078 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.012034    2573 scope.go:117] "RemoveContainer" containerID="09eda01fe729eb1447e1c55a3eb090335843d8d761f357d9f3274f0bb4213e7c"
Jul 17 01:13:09.012320 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.012300    2573 scope.go:117] "RemoveContainer" containerID="acea6139554f9f9d0dc62c1886ff9d217af87b5904d6763b82ab38f207602550"
Jul 17 01:13:09.012520 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:13:09.012495    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh_odh-ai-gateway-infra(5719043e-4fff-4830-be1a-af4c9b34711b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" podUID="5719043e-4fff-4830-be1a-af4c9b34711b"
Jul 17 01:13:09.013890 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.013870    2573 generic.go:356] "Generic (PLEG): container finished" podID="2d809380-837e-471e-a480-029ec0aa80c1" containerID="05b95a747a9ba26abf6ae1ec112eade620ae58b84db77e3aa0ac3eed90081706" exitCode=6
Jul 17 01:13:09.013982 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.013949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" event={"ID":"2d809380-837e-471e-a480-029ec0aa80c1","Type":"ContainerDied","Data":"05b95a747a9ba26abf6ae1ec112eade620ae58b84db77e3aa0ac3eed90081706"}
Jul 17 01:13:09.014211 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.014198    2573 scope.go:117] "RemoveContainer" containerID="05b95a747a9ba26abf6ae1ec112eade620ae58b84db77e3aa0ac3eed90081706"
Jul 17 01:13:09.014355 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:13:09.014338    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-srzxz_odh-ai-gateway-infra(2d809380-837e-471e-a480-029ec0aa80c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" podUID="2d809380-837e-471e-a480-029ec0aa80c1"
Jul 17 01:13:09.024190 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.024116    2573 scope.go:117] "RemoveContainer" containerID="00b96269e5eeb394afa033eabcefca556580c267f71dfb13e3b9a3c9e4129eb2"
Jul 17 01:13:09.148326 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.148083    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pfsxc_fb595638-7449-478a-8505-2e0e531829a1/global-pull-secret-syncer/0.log"
Jul 17 01:13:09.290662 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.290642    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zxwfp_7f892710-cf20-4bdb-aa0b-0c42e8f9436a/konnectivity-agent/0.log"
Jul 17 01:13:09.335312 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:09.335291    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-41.ec2.internal_002d47e5af4c93f026b6cc20e8b1a908/haproxy/0.log"
Jul 17 01:13:10.021830 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.021777    2573 generic.go:356] "Generic (PLEG): container finished" podID="85714b14-cc44-4b5f-aa80-14402fb49423" containerID="c6e81c1bb6ffd21718b8838d772484d3bb1825bb1d0601145db00a7e0d183222" exitCode=6
Jul 17 01:13:10.022258 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.021851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" event={"ID":"85714b14-cc44-4b5f-aa80-14402fb49423","Type":"ContainerDied","Data":"c6e81c1bb6ffd21718b8838d772484d3bb1825bb1d0601145db00a7e0d183222"}
Jul 17 01:13:10.022258 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.021915    2573 scope.go:117] "RemoveContainer" containerID="5c4ee6333e112c87f59b4fa8e06e77b82fb0ff811cae65772e96f559bb0d5a15"
Jul 17 01:13:10.022258 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.022197    2573 scope.go:117] "RemoveContainer" containerID="c6e81c1bb6ffd21718b8838d772484d3bb1825bb1d0601145db00a7e0d183222"
Jul 17 01:13:10.022411 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:13:10.022388    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt_odh-ai-gateway-infra(85714b14-cc44-4b5f-aa80-14402fb49423)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" podUID="85714b14-cc44-4b5f-aa80-14402fb49423"
Jul 17 01:13:10.036095 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.036068    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"]
Jul 17 01:13:10.046814 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.046776    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"]
Jul 17 01:13:10.174591 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.174569    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"
Jul 17 01:13:10.207065 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.207041    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"
Jul 17 01:13:10.275758 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.275704    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d809380-837e-471e-a480-029ec0aa80c1-maas-api-service-ca\") pod \"2d809380-837e-471e-a480-029ec0aa80c1\" (UID: \"2d809380-837e-471e-a480-029ec0aa80c1\") "
Jul 17 01:13:10.275877 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.275834    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5719043e-4fff-4830-be1a-af4c9b34711b-maas-api-service-ca\") pod \"5719043e-4fff-4830-be1a-af4c9b34711b\" (UID: \"5719043e-4fff-4830-be1a-af4c9b34711b\") "
Jul 17 01:13:10.276131 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.276107    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d809380-837e-471e-a480-029ec0aa80c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d809380-837e-471e-a480-029ec0aa80c1" (UID: "2d809380-837e-471e-a480-029ec0aa80c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:13:10.276175 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.276161    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5719043e-4fff-4830-be1a-af4c9b34711b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5719043e-4fff-4830-be1a-af4c9b34711b" (UID: "5719043e-4fff-4830-be1a-af4c9b34711b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:13:10.377183 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.377160    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5719043e-4fff-4830-be1a-af4c9b34711b-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:13:10.377183 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:10.377182    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d809380-837e-471e-a480-029ec0aa80c1-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:13:11.032849 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.032822    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"
Jul 17 01:13:11.032849 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.032841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh" event={"ID":"5719043e-4fff-4830-be1a-af4c9b34711b","Type":"ContainerDied","Data":"e57e2b4ea8f2bd24c1b26591a0394986b6819cdcd7818333000b82f4ec7c030b"}
Jul 17 01:13:11.033341 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.032898    2573 scope.go:117] "RemoveContainer" containerID="acea6139554f9f9d0dc62c1886ff9d217af87b5904d6763b82ab38f207602550"
Jul 17 01:13:11.034174 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.034150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz" event={"ID":"2d809380-837e-471e-a480-029ec0aa80c1","Type":"ContainerDied","Data":"371ce1f2ec69379051902c2cd78d2d1959fabce663df551c970be12b63a6882a"}
Jul 17 01:13:11.034256 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.034163    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"
Jul 17 01:13:11.041657 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.041635    2573 scope.go:117] "RemoveContainer" containerID="05b95a747a9ba26abf6ae1ec112eade620ae58b84db77e3aa0ac3eed90081706"
Jul 17 01:13:11.048231 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.048212    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"]
Jul 17 01:13:11.058390 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.058368    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"]
Jul 17 01:13:11.066703 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.066682    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-srmwh"]
Jul 17 01:13:11.076809 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.076772    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"]
Jul 17 01:13:11.081833 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.081812    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-srzxz"]
Jul 17 01:13:11.176526 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.176505    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"
Jul 17 01:13:11.283104 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.283041    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85714b14-cc44-4b5f-aa80-14402fb49423-maas-api-service-ca\") pod \"85714b14-cc44-4b5f-aa80-14402fb49423\" (UID: \"85714b14-cc44-4b5f-aa80-14402fb49423\") "
Jul 17 01:13:11.283338 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.283317    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85714b14-cc44-4b5f-aa80-14402fb49423-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85714b14-cc44-4b5f-aa80-14402fb49423" (UID: "85714b14-cc44-4b5f-aa80-14402fb49423"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 01:13:11.384376 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.384350    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85714b14-cc44-4b5f-aa80-14402fb49423-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Jul 17 01:13:11.967826 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.967775    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d809380-837e-471e-a480-029ec0aa80c1" path="/var/lib/kubelet/pods/2d809380-837e-471e-a480-029ec0aa80c1/volumes"
Jul 17 01:13:11.968127 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:11.968113    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" path="/var/lib/kubelet/pods/5719043e-4fff-4830-be1a-af4c9b34711b/volumes"
Jul 17 01:13:12.041615 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:12.041594    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"
Jul 17 01:13:12.042030 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:12.041611    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt" event={"ID":"85714b14-cc44-4b5f-aa80-14402fb49423","Type":"ContainerDied","Data":"5deef50b7fb439d55209705777dc8230f23542a0b6215c1eddd11b5cbee29d7d"}
Jul 17 01:13:12.042030 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:12.041657    2573 scope.go:117] "RemoveContainer" containerID="c6e81c1bb6ffd21718b8838d772484d3bb1825bb1d0601145db00a7e0d183222"
Jul 17 01:13:12.056017 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:12.055996    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"]
Jul 17 01:13:12.060362 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:12.060339    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-f28mt"]
Jul 17 01:13:13.917375 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:13.917336    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc_db088276-f18e-4fd4-ab03-8fb01725451e/extract/0.log"
Jul 17 01:13:13.937404 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:13.937380    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc_db088276-f18e-4fd4-ab03-8fb01725451e/util/0.log"
Jul 17 01:13:13.957502 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:13.957479    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pzxpc_db088276-f18e-4fd4-ab03-8fb01725451e/pull/0.log"
Jul 17 01:13:13.968204 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:13.968180    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" path="/var/lib/kubelet/pods/85714b14-cc44-4b5f-aa80-14402fb49423/volumes"
Jul 17 01:13:13.982909 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:13.982877    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp_2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41/extract/0.log"
Jul 17 01:13:14.002160 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.002140    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp_2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41/util/0.log"
Jul 17 01:13:14.021872 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.021850    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06h2dp_2aece1f6-a8c2-4eaa-948b-4f1e0bd20a41/pull/0.log"
Jul 17 01:13:14.050786 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.050765    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk_84c1c8fb-4cef-450e-aa29-69685b2384e3/extract/0.log"
Jul 17 01:13:14.072280 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.072255    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk_84c1c8fb-4cef-450e-aa29-69685b2384e3/util/0.log"
Jul 17 01:13:14.091876 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.091857    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739t7mk_84c1c8fb-4cef-450e-aa29-69685b2384e3/pull/0.log"
Jul 17 01:13:14.117021 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.117002    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq_1b84bdb2-14fe-4da0-b015-f5c40e62a2e8/extract/0.log"
Jul 17 01:13:14.137132 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.137112    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq_1b84bdb2-14fe-4da0-b015-f5c40e62a2e8/util/0.log"
Jul 17 01:13:14.157048 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.157031    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xl7qq_1b84bdb2-14fe-4da0-b015-f5c40e62a2e8/pull/0.log"
Jul 17 01:13:14.734293 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.734267    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gr8xz_f7b46a10-a42d-40b5-8ad6-986d6740be80/manager/0.log"
Jul 17 01:13:14.758350 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.758325    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rjmbk_750eefa1-5519-4cb9-bebc-58e9d2a55d48/manager/0.log"
Jul 17 01:13:14.782518 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.782499    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kszxw_c02072b4-cf4f-4566-8fc0-e5b552507e54/kuadrant-console-plugin/0.log"
Jul 17 01:13:14.836591 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:14.836570    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6mc2k_cbc87a2e-6674-4bb4-ae2e-018e6362a9a6/registry-server/0.log"
Jul 17 01:13:15.029908 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:15.029868    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bjvkl_45577182-8092-4ac7-b438-cbb85b50724a/manager/0.log"
Jul 17 01:13:17.149462 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:17.149435    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5fdc7b5777-jfptx_06aaf4d6-823e-416f-a28a-29877e7c8223/metrics-server/0.log"
Jul 17 01:13:17.176941 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:17.176916    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-8tnv5_9b9f177b-9904-4f5a-9610-8c9d79745ad8/monitoring-plugin/0.log"
Jul 17 01:13:17.275470 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:17.275451    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f99ft_54fc07a8-3185-4927-b5d9-8e39e973f51d/node-exporter/0.log"
Jul 17 01:13:17.303626 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:17.303599    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f99ft_54fc07a8-3185-4927-b5d9-8e39e973f51d/kube-rbac-proxy/0.log"
Jul 17 01:13:17.324477 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:17.324463    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f99ft_54fc07a8-3185-4927-b5d9-8e39e973f51d/init-textfile/0.log"
Jul 17 01:13:17.714960 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:17.714939    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-snsws_a26c5aaa-9bc1-4a94-8687-386cb8004934/prometheus-operator/0.log"
Jul 17 01:13:17.738083 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:17.738066    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-snsws_a26c5aaa-9bc1-4a94-8687-386cb8004934/kube-rbac-proxy/0.log"
Jul 17 01:13:19.479749 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.479719    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"]
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480207    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480224    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480234    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480240    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480255    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480260    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480266    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:19.480273 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480271    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480282    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480287    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480296    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480301    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480357    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480363    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480369    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480376    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480380    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:19.480552 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.480387    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:19.484832 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.484809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.486695 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.486670    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gf5nn\"/\"kube-root-ca.crt\""
Jul 17 01:13:19.486841 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.486773    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gf5nn\"/\"default-dockercfg-qjfmp\""
Jul 17 01:13:19.487579 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.487562    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gf5nn\"/\"openshift-service-ca.crt\""
Jul 17 01:13:19.493259 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.493234    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"]
Jul 17 01:13:19.643770 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.643746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-proc\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.643910 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.643801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-lib-modules\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.643959 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.643898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6j2d\" (UniqueName: \"kubernetes.io/projected/aff67a18-f9f6-4046-8d37-294c9ec22252-kube-api-access-j6j2d\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.643959 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.643940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-sys\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.644047 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.643983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-podres\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744801 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-podres\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744801 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-proc\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-lib-modules\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6j2d\" (UniqueName: \"kubernetes.io/projected/aff67a18-f9f6-4046-8d37-294c9ec22252-kube-api-access-j6j2d\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-sys\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744898    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-podres\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744904    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-proc\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.744971 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-sys\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.745190 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.744986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aff67a18-f9f6-4046-8d37-294c9ec22252-lib-modules\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.752028 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.752009    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6j2d\" (UniqueName: \"kubernetes.io/projected/aff67a18-f9f6-4046-8d37-294c9ec22252-kube-api-access-j6j2d\") pod \"perf-node-gather-daemonset-nw4pn\" (UID: \"aff67a18-f9f6-4046-8d37-294c9ec22252\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.795831 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.795812    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:19.922195 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:19.922163    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"]
Jul 17 01:13:19.923279 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:13:19.923241    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaff67a18_f9f6_4046_8d37_294c9ec22252.slice/crio-ef1a93de298665abcd61b25db117d2a441506de909b01c57fee7fa4c748d6a0c WatchSource:0}: Error finding container ef1a93de298665abcd61b25db117d2a441506de909b01c57fee7fa4c748d6a0c: Status 404 returned error can't find the container with id ef1a93de298665abcd61b25db117d2a441506de909b01c57fee7fa4c748d6a0c
Jul 17 01:13:20.085840 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:20.085813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn" event={"ID":"aff67a18-f9f6-4046-8d37-294c9ec22252","Type":"ContainerStarted","Data":"e03dd58e746e7a81607ef7b403082744d766b720be2bfe3f2bd7b742b3b548d5"}
Jul 17 01:13:20.085984 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:20.085868    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:20.085984 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:20.085899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn" event={"ID":"aff67a18-f9f6-4046-8d37-294c9ec22252","Type":"ContainerStarted","Data":"ef1a93de298665abcd61b25db117d2a441506de909b01c57fee7fa4c748d6a0c"}
Jul 17 01:13:20.100572 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:20.100528    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn" podStartSLOduration=1.100514246 podStartE2EDuration="1.100514246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:13:20.098606764 +0000 UTC m=+3394.674759337" watchObservedRunningTime="2026-07-17 01:13:20.100514246 +0000 UTC m=+3394.676666859"
Jul 17 01:13:20.244074 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:20.244043    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-79bfbfc896-hk6zb_bf073599-1ef5-4774-88fd-c13a7d50266d/console/0.log"
Jul 17 01:13:20.283786 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:20.283763    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-skjmc_0efef89c-53a3-4a48-ba70-529242dad2b8/download-server/0.log"
Jul 17 01:13:21.936368 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:21.936340    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zxczp_74ed1822-52df-4463-87a9-cd5367dfca3a/dns/0.log"
Jul 17 01:13:21.955533 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:21.955511    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zxczp_74ed1822-52df-4463-87a9-cd5367dfca3a/kube-rbac-proxy/0.log"
Jul 17 01:13:22.003525 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:22.003505    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6x662_03a2ee30-6f7d-438c-bac7-52add958ae4c/dns-node-resolver/0.log"
Jul 17 01:13:22.542503 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:22.542479    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c97cc46cb-jmhcr_8f9d1891-b7c3-42a4-951b-4145be0e8ffb/registry/0.log"
Jul 17 01:13:22.609051 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:22.609029    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rxcmn_d28d955d-d569-4155-8732-247fd315be8f/node-ca/0.log"
Jul 17 01:13:23.743705 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:23.743670    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jndrl_48e44104-5be3-4c4f-b021-e22713374ecf/discovery/0.log"
Jul 17 01:13:23.766420 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:23.766387    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-75nvg_85e79f05-6be2-4c22-b098-89eb2450e25c/kube-auth-proxy/0.log"
Jul 17 01:13:23.916873 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:23.916846    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pm4jz_af6b4198-05cf-4598-a670-096a6f7cf460/payload-pre-processing/0.log"
Jul 17 01:13:24.713118 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:24.713089    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ph4x4_863459a2-790d-4fab-afd5-b9a255285b3a/serve-healthcheck-canary/0.log"
Jul 17 01:13:25.270238 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:25.270213    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xkcbl_78bc1249-e4fb-49bd-8d71-ba85d657c82d/kube-rbac-proxy/0.log"
Jul 17 01:13:25.292057 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:25.292033    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xkcbl_78bc1249-e4fb-49bd-8d71-ba85d657c82d/exporter/0.log"
Jul 17 01:13:25.313159 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:25.313134    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xkcbl_78bc1249-e4fb-49bd-8d71-ba85d657c82d/extractor/0.log"
Jul 17 01:13:26.100509 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.100486    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-nw4pn"
Jul 17 01:13:26.817898 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.817863    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"]
Jul 17 01:13:26.818256 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818244    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:26.818256 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818255    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:26.818335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818269    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:26.818335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818275    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:26.818335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818285    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:26.818335 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818291    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:26.818461 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818356    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d809380-837e-471e-a480-029ec0aa80c1" containerName="revoke-keys"
Jul 17 01:13:26.818461 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818367    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5719043e-4fff-4830-be1a-af4c9b34711b" containerName="revoke-keys"
Jul 17 01:13:26.818461 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.818373    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="85714b14-cc44-4b5f-aa80-14402fb49423" containerName="revoke-keys"
Jul 17 01:13:26.821477 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.821461    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"
Jul 17 01:13:26.823448 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.823418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\""
Jul 17 01:13:26.826969 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:26.826947    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"]
Jul 17 01:13:27.000938 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.000911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2da013-ae9c-432e-b1b4-b3f3ddab6105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-hjg4q\" (UID: \"8e2da013-ae9c-432e-b1b4-b3f3ddab6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"
Jul 17 01:13:27.102064 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.101982    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2da013-ae9c-432e-b1b4-b3f3ddab6105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-hjg4q\" (UID: \"8e2da013-ae9c-432e-b1b4-b3f3ddab6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"
Jul 17 01:13:27.102489 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.102473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2da013-ae9c-432e-b1b4-b3f3ddab6105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-hjg4q\" (UID: \"8e2da013-ae9c-432e-b1b4-b3f3ddab6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"
Jul 17 01:13:27.133869 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.133849    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"
Jul 17 01:13:27.260287 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.260254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q"]
Jul 17 01:13:27.261367 ip-10-0-139-41 kubenswrapper[2573]: W0717 01:13:27.261345    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2da013_ae9c_432e_b1b4_b3f3ddab6105.slice/crio-886678b4ee745f52af53dd17fcf7b57eae2b9aa1c06d2aed453d51a7d0b4c621 WatchSource:0}: Error finding container 886678b4ee745f52af53dd17fcf7b57eae2b9aa1c06d2aed453d51a7d0b4c621: Status 404 returned error can't find the container with id 886678b4ee745f52af53dd17fcf7b57eae2b9aa1c06d2aed453d51a7d0b4c621
Jul 17 01:13:27.784610 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.784581    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-965b4997-zn7mh_a21cd433-0636-49ed-871e-67a383ac3c87/manager/0.log"
Jul 17 01:13:27.802907 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.802887    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xvwgt_7a35fd16-bf16-46a5-a26a-8e3d83d0b66b/server/0.log"
Jul 17 01:13:27.902236 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:27.902209    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v92p8_8b2d5d3c-64f4-4d5c-a629-60819a1262a6/manager/0.log"
Jul 17 01:13:28.034641 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:28.034585    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-8mc6b_47849ce6-96d2-4b16-9ce6-9b6765a6b5ed/manager/0.log"
Jul 17 01:13:28.051308 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:28.051287    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-8mc6b_47849ce6-96d2-4b16-9ce6-9b6765a6b5ed/copy-manifests/0.log"
Jul 17 01:13:28.131418 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:28.131394    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e2da013-ae9c-432e-b1b4-b3f3ddab6105" containerID="a1587961b9f90f54c981a45a22589e076af44ebc312c62923edae7939489f82d" exitCode=6
Jul 17 01:13:28.131521 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:28.131478    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q" event={"ID":"8e2da013-ae9c-432e-b1b4-b3f3ddab6105","Type":"ContainerDied","Data":"a1587961b9f90f54c981a45a22589e076af44ebc312c62923edae7939489f82d"}
Jul 17 01:13:28.131575 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:28.131526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q" event={"ID":"8e2da013-ae9c-432e-b1b4-b3f3ddab6105","Type":"ContainerStarted","Data":"886678b4ee745f52af53dd17fcf7b57eae2b9aa1c06d2aed453d51a7d0b4c621"}
Jul 17 01:13:28.131610 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:28.131581    2573 scope.go:117] "RemoveContainer" containerID="a1587961b9f90f54c981a45a22589e076af44ebc312c62923edae7939489f82d"
Jul 17 01:13:29.138699 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:29.138668    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e2da013-ae9c-432e-b1b4-b3f3ddab6105" containerID="d4113de5f9c1b8b7a782fbd5b1a1fc49c626bc615edd2e1dcecbc6ac42407a13" exitCode=6
Jul 17 01:13:29.139095 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:29.138722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q" event={"ID":"8e2da013-ae9c-432e-b1b4-b3f3ddab6105","Type":"ContainerDied","Data":"d4113de5f9c1b8b7a782fbd5b1a1fc49c626bc615edd2e1dcecbc6ac42407a13"}
Jul 17 01:13:29.139095 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:29.138766    2573 scope.go:117] "RemoveContainer" containerID="a1587961b9f90f54c981a45a22589e076af44ebc312c62923edae7939489f82d"
Jul 17 01:13:29.139095 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:29.138896    2573 scope.go:117] "RemoveContainer" containerID="d4113de5f9c1b8b7a782fbd5b1a1fc49c626bc615edd2e1dcecbc6ac42407a13"
Jul 17 01:13:29.139095 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:13:29.139085    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-hjg4q_odh-ai-gateway-infra(8e2da013-ae9c-432e-b1b4-b3f3ddab6105)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q" podUID="8e2da013-ae9c-432e-b1b4-b3f3ddab6105"
Jul 17 01:13:29.476332 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:29.476305    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jndrl_48e44104-5be3-4c4f-b021-e22713374ecf/discovery/0.log"
Jul 17 01:13:29.487154 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:29.487132    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-75nvg_85e79f05-6be2-4c22-b098-89eb2450e25c/kube-auth-proxy/0.log"
Jul 17 01:13:29.623812 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:29.623771    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pm4jz_af6b4198-05cf-4598-a670-096a6f7cf460/payload-pre-processing/0.log"
Jul 17 01:13:30.145920 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:30.145885    2573 scope.go:117] "RemoveContainer" containerID="d4113de5f9c1b8b7a782fbd5b1a1fc49c626bc615edd2e1dcecbc6ac42407a13"
Jul 17 01:13:30.146367 ip-10-0-139-41 kubenswrapper[2573]: E0717 01:13:30.146115    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-hjg4q_odh-ai-gateway-infra(8e2da013-ae9c-432e-b1b4-b3f3ddab6105)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hjg4q" podUID="8e2da013-ae9c-432e-b1b4-b3f3ddab6105"
Jul 17 01:13:31.221368 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:31.221336    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jndrl_48e44104-5be3-4c4f-b021-e22713374ecf/discovery/0.log"
Jul 17 01:13:31.232701 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:31.232673    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-75nvg_85e79f05-6be2-4c22-b098-89eb2450e25c/kube-auth-proxy/0.log"
Jul 17 01:13:31.359358 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:31.359331    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pm4jz_af6b4198-05cf-4598-a670-096a6f7cf460/payload-pre-processing/0.log"
Jul 17 01:13:35.803349 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:35.803320    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8pdf_00c347f5-1f2b-402c-8edf-cc04d4832887/kube-multus-additional-cni-plugins/0.log"
Jul 17 01:13:35.822685 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:35.822654    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8pdf_00c347f5-1f2b-402c-8edf-cc04d4832887/egress-router-binary-copy/0.log"
Jul 17 01:13:35.841618 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:35.841588    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8pdf_00c347f5-1f2b-402c-8edf-cc04d4832887/cni-plugins/0.log"
Jul 17 01:13:35.860413 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:35.860390    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8pdf_00c347f5-1f2b-402c-8edf-cc04d4832887/bond-cni-plugin/0.log"
Jul 17 01:13:35.879151 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:35.879133    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8pdf_00c347f5-1f2b-402c-8edf-cc04d4832887/routeoverride-cni/0.log"
Jul 17 01:13:35.897946 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:35.897930    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8pdf_00c347f5-1f2b-402c-8edf-cc04d4832887/whereabouts-cni-bincopy/0.log"
Jul 17 01:13:35.917485 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:35.917467    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8pdf_00c347f5-1f2b-402c-8edf-cc04d4832887/whereabouts-cni/0.log"
Jul 17 01:13:36.132625 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:36.132553    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cdpqd_9662c6e8-79c3-4f03-b1bf-e9f85cac2207/kube-multus/0.log"
Jul 17 01:13:36.311866 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:36.311836    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vjwrp_7e1bc817-d1b9-458a-a933-8af8ce5e95b3/network-metrics-daemon/0.log"
Jul 17 01:13:36.326749 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:36.326724    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vjwrp_7e1bc817-d1b9-458a-a933-8af8ce5e95b3/kube-rbac-proxy/0.log"
Jul 17 01:13:37.381506 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.381473    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-controller/0.log"
Jul 17 01:13:37.398134 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.398108    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/0.log"
Jul 17 01:13:37.398284 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.398199    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovn-acl-logging/1.log"
Jul 17 01:13:37.417348 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.417326    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/kube-rbac-proxy-node/0.log"
Jul 17 01:13:37.435864 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.435849    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 17 01:13:37.456198 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.456179    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/northd/0.log"
Jul 17 01:13:37.475468 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.475445    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/nbdb/0.log"
Jul 17 01:13:37.494832 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.494810    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/sbdb/0.log"
Jul 17 01:13:37.646515 ip-10-0-139-41 kubenswrapper[2573]: I0717 01:13:37.646455    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-645g2_983c692a-3a52-4a5d-b36f-4ded907ffd0d/ovnkube-controller/0.log"