Jul 17 19:56:05.186383 ip-10-0-143-185 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.594124    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601249    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601264    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601268    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601271    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601274    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601279    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601281    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 19:56:05.646344 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601286    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601289    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601292    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601295    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601298    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601301    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601303    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601306    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601308    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601311    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601314    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601316    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601319    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601321    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601324    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601327    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601329    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601333    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601338    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 19:56:05.647639 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601341    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601345    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601349    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601352    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601355    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601358    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601361    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601364    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601366    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601369    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601371    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601374    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601377    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601379    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601382    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601384    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601386    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601389    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601391    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601394    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 19:56:05.648900 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601396    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601399    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601401    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601403    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601406    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601409    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601411    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601414    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601416    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601419    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601421    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601424    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601426    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601429    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601431    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601434    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601437    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601439    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601443    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601446    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 19:56:05.649760 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601448    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601451    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601453    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601456    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601458    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601460    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601464    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601468    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601471    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601474    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601477    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601480    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601482    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601485    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601488    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601491    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601494    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601497    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601500    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 19:56:05.651033 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601503    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601505    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601508    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601510    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601513    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601515    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601518    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601521    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601523    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601526    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601528    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601531    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601534    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.601536    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602237    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602244    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602247    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602249    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602252    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602255    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 19:56:05.651901 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602257    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602262    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602266    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602269    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602272    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602274    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602277    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602280    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602283    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602285    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602288    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602292    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602295    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602297    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602300    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602303    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602305    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602308    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602310    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 19:56:05.652980 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602314    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602316    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602319    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602322    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602325    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602327    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602330    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602333    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602335    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602338    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602341    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602343    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602346    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602348    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602351    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602353    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602356    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602359    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602361    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602364    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 19:56:05.654477 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602367    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602369    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602372    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602374    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602377    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602381    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602384    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602386    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602389    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602391    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602394    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602396    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602399    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602402    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602406    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602412    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602415    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602418    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602420    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602423    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 19:56:05.655268 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602426    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602429    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602432    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602435    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602437    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602440    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602443    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602445    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602447    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602451    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602454    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602456    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602459    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602461    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602464    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602466    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602469    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602471    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602474    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 19:56:05.656102 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602477    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602479    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602482    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602484    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602486    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602489    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602492    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602494    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602499    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602501    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602504    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602506    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602509    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602511    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602513    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602936    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602942    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602946    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602949    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602953    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 19:56:05.656775 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602956    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602959    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602961    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602964    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602967    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602970    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602973    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602975    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602978    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602981    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602985    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602989    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602993    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602995    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.602998    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603001    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603003    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603006    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603009    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 19:56:05.657499 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603012    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603016    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603019    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603021    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603024    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603026    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603029    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603031    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603034    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603036    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603039    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603041    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603061    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603063    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603066    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603069    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603071    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603074    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603076    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603079    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 19:56:05.658186 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603082    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603084    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603087    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603089    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603092    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603095    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603099    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603102    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603105    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603108    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603115    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603118    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603120    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603124    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603127    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603129    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603132    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603134    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603137    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 19:56:05.658899 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603139    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603142    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603144    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603154    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603158    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603161    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603163    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603166    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603169    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603171    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603173    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603176    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603179    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603181    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603183    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603186    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603188    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603191    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603193    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 19:56:05.659584 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603197    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603200    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603203    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603206    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603210    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603212    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603215    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603219    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603222    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603224    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603227    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603229    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603232    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603234    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603237    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603239    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603242    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603305    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603311    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603317    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603322    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 19:56:05.660263 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603326    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603329    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603333    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603337    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603340    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603344    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603347    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603350    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603353    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603356    2572 flags.go:64] FLAG: --cgroup-root=""
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603359    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603362    2572 flags.go:64] FLAG: --client-ca-file=""
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603366    2572 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603369    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603373    2572 flags.go:64] FLAG: --cluster-domain=""
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603378    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603381    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603384    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603388    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603392    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603395    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603398    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603401    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603404    2572 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 19:56:05.660999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603407    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603410    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603413    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603416    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603420    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603423    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603426    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603429    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603432    2572 flags.go:64] FLAG: --enable-server="true"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603435    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603439    2572 flags.go:64] FLAG: --event-burst="100"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603442    2572 flags.go:64] FLAG: --event-qps="50"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603445    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603448    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603451    2572 flags.go:64] FLAG: --eviction-hard=""
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603455    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603458    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603461    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603463    2572 flags.go:64] FLAG: --eviction-soft=""
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603466    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603469    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603472    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603475    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603480    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603483    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 19:56:05.661832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603486    2572 flags.go:64] FLAG: --feature-gates=""
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603490    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603494    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603497    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603499    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603502    2572 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603505    2572 flags.go:64] FLAG: --help="false"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603508    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-143-185.ec2.internal"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603511    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603514    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603517    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603520    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603524    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603527    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603530    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603532    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603535    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603538    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603542    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603545    2572 flags.go:64] FLAG: --kube-reserved=""
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603547    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603550    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603554    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603556    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 19:56:05.662779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603559    2572 flags.go:64] FLAG: --lock-file=""
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603561    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603565    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603568    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603573    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603576    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603581    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603584    2572 flags.go:64] FLAG: --logging-format="text"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603586    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603591    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603594    2572 flags.go:64] FLAG: --manifest-url=""
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603597    2572 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603601    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603604    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603607    2572 flags.go:64] FLAG: --max-pods="110"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603610    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603613    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603616    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603619    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603622    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603625    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603628    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603636    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603639    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603642    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 19:56:05.663654 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603645    2572 flags.go:64] FLAG: --pod-cidr=""
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603648    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603653    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603656    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603659    2572 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603662    2572 flags.go:64] FLAG: --port="10250"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603665    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603668    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0735583504e5b0518"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603671    2572 flags.go:64] FLAG: --qos-reserved=""
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603674    2572 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603677    2572 flags.go:64] FLAG: --register-node="true"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603680    2572 flags.go:64] FLAG: --register-with-taints=""
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603684    2572 flags.go:64] FLAG: --registry-burst="10"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603687    2572 flags.go:64] FLAG: --registry-qps="5"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603691    2572 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603694    2572 flags.go:64] FLAG: --reserved-memory=""
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603699    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603702    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603705    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603708    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603710    2572 flags.go:64] FLAG: --runonce="false"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603713    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603716    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603719    2572 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603722    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 19:56:05.664542 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603725    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603728    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603731    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603734    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603737    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603740    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603743    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603746    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603749    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603752    2572 flags.go:64] FLAG: --system-cgroups=""
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603754    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603760    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603762    2572 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603765    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603768    2572 flags.go:64] FLAG: --tls-min-version=""
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603771    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603774    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603777    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603780    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603782    2572 flags.go:64] FLAG: --v="2"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603786    2572 flags.go:64] FLAG: --version="false"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603792    2572 flags.go:64] FLAG: --vmodule=""
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603796    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.603800    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603910    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 19:56:05.665437 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603914    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603917    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603920    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603923    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603925    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603929    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603932    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603934    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603937    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603940    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603943    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603946    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603948    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603952    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603956    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603959    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603962    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603965    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603968    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 19:56:05.666354 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603970    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603973    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603976    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603979    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603982    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603985    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603987    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603990    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603993    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.603996    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604000    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604004    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604007    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604009    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604012    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604014    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604017    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604019    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604022    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604025    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 19:56:05.667060 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604027    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604030    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604032    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604035    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604038    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604040    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604060    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604087    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604090    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604093    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604096    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604099    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604102    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604104    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604107    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604110    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604113    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604115    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604118    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604120    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 19:56:05.667783 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604123    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604125    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604128    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604132    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604135    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604137    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604140    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604143    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604146    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604148    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604151    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604153    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604156    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604158    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604161    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604163    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604166    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604169    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604172    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604175    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 19:56:05.668538 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604178    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604180    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604183    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604188    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604191    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604194    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604197    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604200    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604203    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604206    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604208    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604211    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604214    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604216    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604220    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604223    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604227    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604229    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 19:56:05.669262 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.604233    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.604864    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.611478    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.611493    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.611512    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.611518    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611575    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611580    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611583    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611587    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611590    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611593    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611596    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611599    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611602    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 19:56:05.669898 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611605    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611608    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611611    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611613    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611616    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611619    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611622    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611625    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611628    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611630    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611633    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611636    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611638    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611642    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611644    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611647    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611650    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611653    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611656    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611658    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 19:56:05.670464 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611662    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611665    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611667    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611670    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611672    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611675    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611677    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611679    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611682    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611684    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611687    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611690    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611692    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611695    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611697    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611700    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611703    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611706    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611708    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 19:56:05.671210 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611711    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611713    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611716    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611718    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611721    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611723    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611726    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611729    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611731    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611735    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611737    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611740    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611742    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611745    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611748    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611751    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611753    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611756    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611760    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611763    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 19:56:05.671885 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611766    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611769    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611772    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611774    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611777    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611780    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611783    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611785    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611788    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611791    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611793    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611797    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611801    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611803    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611806    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611810    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611813    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611815    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611821    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 19:56:05.672602 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611824    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611828    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611832    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611835    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611838    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611841    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611844    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611846    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611849    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611852    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611855    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611858    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.611863    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611975    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611980    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 19:56:05.673300 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611982    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611985    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611988    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611990    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611993    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611995    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.611998    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612001    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612003    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612006    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612008    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612011    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612013    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612016    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612018    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612020    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612025    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612028    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612030    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612033    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 19:56:05.673837 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612037    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612039    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612042    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612062    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612065    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612067    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612071    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612073    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612077    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612081    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612085    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612087    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612090    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612093    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612095    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612098    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612101    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612103    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612107    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 19:56:05.674542 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612110    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612113    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612115    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612117    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612120    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612123    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612125    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612128    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612130    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612135    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612137    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612141    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612144    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612149    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612152    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612155    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612157    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612160    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612163    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612165    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 19:56:05.675244 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612168    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612171    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612173    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612176    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612178    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612181    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612183    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612186    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612188    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612191    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612194    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612196    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612199    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612201    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612204    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612206    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612209    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612211    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612214    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612216    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 19:56:05.675963 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612219    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612223    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612225    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612228    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612230    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612235    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612237    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612240    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612242    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612245    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612247    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612250    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612252    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612255    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612258    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612261    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612263    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 19:56:05.676718 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:05.612266    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.612271    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.612369    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.614256    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.615588    2572 server.go:1019] "Starting client certificate rotation"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.615694    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.615734    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.617987    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.619068    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.631857    2572 log.go:25] "Validated CRI v1 runtime API"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.633684    2572 log.go:25] "Validated CRI v1 image API"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.635162    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 19:56:05.677513 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.637362    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8773c87d-5981-4ca6-8ac1-ab40ea648142:/dev/nvme0n1p4 f5fbbb05-dbf7-4d0a-868e-69d6f4e5dbb2:/dev/nvme0n1p3]
Jul 17 19:56:05.677959 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.637384    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.643943    2572 manager.go:216] Machine: {Timestamp:2026-07-17 19:56:05.642491816 +0000 UTC m=+0.336467973 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107440 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2921c226087b0824eb9ab8ddf00ba0 SystemUUID:ec2921c2-2608-7b08-24eb-9ab8ddf00ba0 BootID:f2e51632-7ed7-404e-b4c6-1d03ef665676 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:24:ce:5c:f2:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:24:ce:5c:f2:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:90:e1:7a:7f:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.644063    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.644136    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.648034    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.648072    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.648829    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.648839    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 19:56:05.678002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.648847    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.649676    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.651098    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.654147    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.654274    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.655021    2572 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.655035    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.655061    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.655078    2572 kubelet.go:398] "Adding apiserver pod source"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.655091    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.657760    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 19:56:05.678357 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.657780    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 19:56:05.681728 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.681708    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 19:56:05.683873 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.683853    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 19:56:05.683965 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.683903    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 19:56:05.685015 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685001    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685020    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685029    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685037    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685063    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685071    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685082    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685091    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685100    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685108    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 19:56:05.685120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685120    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 19:56:05.685398 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.685128    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 19:56:05.686849 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.686838    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 19:56:05.686902 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.686851    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 19:56:05.695029 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.695007    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 19:56:05.695241 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.695218    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 19:56:05.695322 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.695225    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 19:56:05.695322 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.695260    2572 server.go:1268] "Started kubelet"
Jul 17 19:56:05.695401 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.695355    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 19:56:05.699729 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.699705    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 19:56:05.699803 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.699738    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 19:56:05.699846 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.699831    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 19:56:05.700412 ip-10-0-143-185 systemd[1]: Started Kubernetes Kubelet.
Jul 17 19:56:05.702151 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.702131    2572 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 19:56:05.704851 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.704831    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 19:56:05.706121 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.705186    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb008a6f80  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.695229824 +0000 UTC m=+0.389205966,LastTimestamp:2026-07-17 19:56:05.695229824 +0000 UTC m=+0.389205966,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.707425 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.707405    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 19:56:05.717135 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.717117    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 19:56:05.717231 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.717215    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:05.717283 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.717119    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 19:56:05.717926 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.717860    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.718596    2572 factory.go:153] Registering CRI-O factory
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.718610    2572 factory.go:223] Registration of the crio container factory successfully
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.718660    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.718670    2572 factory.go:55] Registering systemd factory
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.718678    2572 factory.go:223] Registration of the systemd container factory successfully
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.718701    2572 factory.go:103] Registering Raw factory
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.718716    2572 manager.go:1196] Started watching for new ooms in manager
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.719113    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 19:56:05.719140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.719133    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 19:56:05.719593 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.719118    2572 manager.go:349] Starting recovery of all containers
Jul 17 19:56:05.720263 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.720250    2572 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 19:56:05.720375 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.720360    2572 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 19:56:05.726735 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.726718    2572 manager.go:354] Recovery completed
Jul 17 19:56:05.727886 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.727859    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 19:56:05.727997 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.727958    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 19:56:05.731623 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.731608    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:05.733333 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.733316    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:05.733399 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.733346    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:05.733399 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.733356    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:05.733909 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.733897    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:05.733980 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.733901    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 19:56:05.733980 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.733962    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 19:56:05.734090 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.733981    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 19:56:05.734738 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.734725    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:05.734779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.734754    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:05.734779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.734765    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:05.735254 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.735240    2572 policy_none.go:49] "None policy: Start"
Jul 17 19:56:05.735297 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.735256    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 19:56:05.735297 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.735266    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 19:56:05.736450 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.736440    2572 policy_none.go:47] "Start"
Jul 17 19:56:05.742182 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.742119    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb02cfd31c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-185.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.73333174 +0000 UTC m=+0.427307879,LastTimestamp:2026-07-17 19:56:05.73333174 +0000 UTC m=+0.427307879,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.749503 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.749441    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb02d01e43  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-185.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.733350979 +0000 UTC m=+0.427327116,LastTimestamp:2026-07-17 19:56:05.733350979 +0000 UTC m=+0.427327116,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.752062 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.751993    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb02d041ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-185.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.733360109 +0000 UTC m=+0.427336246,LastTimestamp:2026-07-17 19:56:05.733360109 +0000 UTC m=+0.427336246,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.763107 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.762996    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-185.ec2.internal.18c32beb02cfd31c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb02cfd31c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-185.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.73333174 +0000 UTC m=+0.427307879,LastTimestamp:2026-07-17 19:56:05.734740046 +0000 UTC m=+0.428716184,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.771316 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.771260    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-185.ec2.internal.18c32beb02d01e43\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb02d01e43  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-185.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.733350979 +0000 UTC m=+0.427327116,LastTimestamp:2026-07-17 19:56:05.734759518 +0000 UTC m=+0.428735654,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.776620 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.776606    2572 manager.go:337] "Starting Device Plugin manager"
Jul 17 19:56:05.776677 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.776642    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 19:56:05.776677 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.776651    2572 server.go:87] "Starting device plugin registration server"
Jul 17 19:56:05.776906 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.776888    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 19:56:05.776982 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.776901    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 19:56:05.777031 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.776993    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 19:56:05.777093 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.777077    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 19:56:05.777093 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.777089    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 19:56:05.777717 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.777589    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 19:56:05.777787 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.777743    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:05.783670 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.783615    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-185.ec2.internal.18c32beb02d041ed\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb02d041ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-185.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.733360109 +0000 UTC m=+0.427336246,LastTimestamp:2026-07-17 19:56:05.734768963 +0000 UTC m=+0.428745100,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.788665 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.788647    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm24z"
Jul 17 19:56:05.794673 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.794619    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-185.ec2.internal.18c32beb0585129f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-185.ec2.internal,UID:ip-10-0-143-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-185.ec2.internal,},FirstTimestamp:2026-07-17 19:56:05.778764447 +0000 UTC m=+0.472740576,LastTimestamp:2026-07-17 19:56:05.778764447 +0000 UTC m=+0.472740576,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-185.ec2.internal,}"
Jul 17 19:56:05.814431 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.807182    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm24z"
Jul 17 19:56:05.846077 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.846048    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 19:56:05.847282 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.847261    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 19:56:05.847363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.847297    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 19:56:05.847363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.847321    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 19:56:05.847363 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.847353    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 19:56:05.858358 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.858340    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 19:56:05.877096 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.877075    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:05.877944 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.877929    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:05.878011 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.877958    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:05.878011 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.877969    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:05.878011 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.877992    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-185.ec2.internal"
Jul 17 19:56:05.895745 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.895723    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-185.ec2.internal"
Jul 17 19:56:05.895831 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.895748    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-185.ec2.internal\": node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:05.947912 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.947837    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal"]
Jul 17 19:56:05.947912 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.947900    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:05.949565 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.949549    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:05.949653 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.949580    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:05.949653 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.949590    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:05.950838 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.950824    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:05.951007 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.950992    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:05.951046 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.951026    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:05.951640 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.951622    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:05.951734 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.951649    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:05.951734 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.951627    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:05.951734 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.951664    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:05.951734 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.951689    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:05.951734 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.951706    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:05.953003 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.952985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:05.953102 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.953011    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:05.953688 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.953672    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:05.953766 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.953699    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:05.953766 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:05.953709    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:05.966473 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.966454    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:05.977930 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.977904    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-185.ec2.internal\" not found" node="ip-10-0-143-185.ec2.internal"
Jul 17 19:56:05.982542 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:05.982524    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-185.ec2.internal\" not found" node="ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.022984 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.022957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee423c54644a9085fb43b1dbbd566c9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-185.ec2.internal\" (UID: \"fee423c54644a9085fb43b1dbbd566c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.023104 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.022989    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6594032f970e6bed2e3d6e0abc512be7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal\" (UID: \"6594032f970e6bed2e3d6e0abc512be7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.023104 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.023008    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6594032f970e6bed2e3d6e0abc512be7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal\" (UID: \"6594032f970e6bed2e3d6e0abc512be7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.067255 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.067229    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.124172 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.124144    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6594032f970e6bed2e3d6e0abc512be7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal\" (UID: \"6594032f970e6bed2e3d6e0abc512be7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.124172 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.124160    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6594032f970e6bed2e3d6e0abc512be7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal\" (UID: \"6594032f970e6bed2e3d6e0abc512be7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.124280 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.124187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6594032f970e6bed2e3d6e0abc512be7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal\" (UID: \"6594032f970e6bed2e3d6e0abc512be7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.124280 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.124206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee423c54644a9085fb43b1dbbd566c9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-185.ec2.internal\" (UID: \"fee423c54644a9085fb43b1dbbd566c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.124280 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.124245    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee423c54644a9085fb43b1dbbd566c9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-185.ec2.internal\" (UID: \"fee423c54644a9085fb43b1dbbd566c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.124375 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.124290    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6594032f970e6bed2e3d6e0abc512be7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal\" (UID: \"6594032f970e6bed2e3d6e0abc512be7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.167722 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.167688    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.270749 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.270680    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.281645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.281627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.286065 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.285895    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:06.373328 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.373287    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.477295 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.477258    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.580020 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.579951    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.617094 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.617044    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 19:56:06.617249 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.617198    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 19:56:06.627004 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.626985    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 19:56:06.680911 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.680883    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.700642 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.700622    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 19:56:06.781792 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.781759    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.811948 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.811898    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 19:51:06 +0000 UTC" deadline="2028-04-01 00:08:12.507107486 +0000 UTC"
Jul 17 19:56:06.811948 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.811940    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14956h12m5.695171523s"
Jul 17 19:56:06.836994 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:06.836935    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594032f970e6bed2e3d6e0abc512be7.slice/crio-7a57b981d6c6ff743e7cd68539876eb61d209092fa27872a3b747c79282d5fea WatchSource:0}: Error finding container 7a57b981d6c6ff743e7cd68539876eb61d209092fa27872a3b747c79282d5fea: Status 404 returned error can't find the container with id 7a57b981d6c6ff743e7cd68539876eb61d209092fa27872a3b747c79282d5fea
Jul 17 19:56:06.844421 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.844395    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:56:06.852366 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.851919    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal" event={"ID":"6594032f970e6bed2e3d6e0abc512be7","Type":"ContainerStarted","Data":"7a57b981d6c6ff743e7cd68539876eb61d209092fa27872a3b747c79282d5fea"}
Jul 17 19:56:06.864473 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:06.864451    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee423c54644a9085fb43b1dbbd566c9.slice/crio-c54d65f1195186da0883e5449c0d2e7cd158a8ee5c1be9b41645a71dd11c04aa WatchSource:0}: Error finding container c54d65f1195186da0883e5449c0d2e7cd158a8ee5c1be9b41645a71dd11c04aa: Status 404 returned error can't find the container with id c54d65f1195186da0883e5449c0d2e7cd158a8ee5c1be9b41645a71dd11c04aa
Jul 17 19:56:06.878476 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.878452    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 19:56:06.879732 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.879712    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 19:56:06.879810 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.879745    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 19:56:06.879810 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:06.879757    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeHasSufficientPID"
Jul 17 19:56:06.881965 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.881947    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:06.985913 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:06.985870    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-185.ec2.internal\" not found"
Jul 17 19:56:07.041611 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.041588    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 19:56:07.119109 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.119031    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:07.132082 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.132062    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 19:56:07.132841 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.132827    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal"
Jul 17 19:56:07.142128 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.142106    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 19:56:07.532689 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.532603    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 19:56:07.656729 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.656694    2572 apiserver.go:52] "Watching apiserver"
Jul 17 19:56:07.663067 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.663038    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 19:56:07.663444 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.663416    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-b5zcw","openshift-network-operator/iptables-alerter-ghpwh","kube-system/global-pull-secret-syncer-mdlgw","kube-system/konnectivity-agent-zxldh","kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal","openshift-network-diagnostics/network-check-target-stgxg","openshift-ovn-kubernetes/ovnkube-node-9tgrl","openshift-cluster-node-tuning-operator/tuned-58hlf","openshift-dns/node-resolver-5z8z9","openshift-image-registry/node-ca-pjbwc","openshift-multus/multus-additional-cni-plugins-jt22m","openshift-multus/multus-vjss5"]
Jul 17 19:56:07.665025 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.665002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.666801 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.666778    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.667340 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.667309    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ntsh6\""
Jul 17 19:56:07.667485 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.667468    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 19:56:07.667560 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.667471    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 19:56:07.668134 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.668028    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.668269 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.668127    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:07.669113 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.669092    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 19:56:07.669240 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.669217    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pw6p2\""
Jul 17 19:56:07.669720 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.669325    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 19:56:07.669720 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.669639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:07.671208 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.671043    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.672614 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.671857    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 19:56:07.672614 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.671911    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 19:56:07.672614 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.672139    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m2dmj\""
Jul 17 19:56:07.672614 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.672512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.673678 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.673659    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 19:56:07.673678 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.673693    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ms2s8\""
Jul 17 19:56:07.674338 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.673932    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 19:56:07.674338 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.674024    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 19:56:07.674338 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.674047    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 19:56:07.674510 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.674489    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.675555 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.675534    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 19:56:07.675669 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.675561    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 19:56:07.675709 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.675672    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lfk7w\""
Jul 17 19:56:07.675775 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.675754    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 19:56:07.676509 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.676488    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.676967 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.676950    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 19:56:07.677059 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.676974    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 19:56:07.677450 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.677284    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 19:56:07.677570 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.677551    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 19:56:07.677705 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.677686    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 19:56:07.678218 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.678194    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.679934 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.679912    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qqzn4\""
Jul 17 19:56:07.680694 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.680353    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 19:56:07.680803 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.680761    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 19:56:07.680870 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.680858    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6hggz\""
Jul 17 19:56:07.681189 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.681171    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 19:56:07.681751 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.681270    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 19:56:07.681751 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.681714    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pd9f4\""
Jul 17 19:56:07.681977 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.681957    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 19:56:07.683100 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.682547    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:07.683100 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.682631    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:07.683687 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.683660    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 19:56:07.684248 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.684225    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:07.684346 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.684310    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:07.686370 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.686253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.688450 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.688425    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rmdcj\""
Jul 17 19:56:07.688618 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.688591    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 19:56:07.688700 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.688684    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 19:56:07.688968 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.688951    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 19:56:07.718516 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.718492    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 19:56:07.719683 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.719665    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 19:56:07.732641 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732614    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 19:56:07.732745 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-device-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.732745 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysctl-conf\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.732745 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-daemon-config\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.732923 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732780    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:07.732923 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.732923 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732854    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovn-node-metrics-cert\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.732923 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-cnibin\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.732923 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e158eee-6883-41ae-996a-5dad5e07629b-serviceca\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732932    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.732973    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-multus-certs\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733007    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9hm\" (UniqueName: \"kubernetes.io/projected/bf6b5042-e40a-4549-a761-2ef68ad8c71b-kube-api-access-qw9hm\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733039    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be9e287a-187f-49b2-aafd-75205fcdff77-agent-certs\") pod \"konnectivity-agent-zxldh\" (UID: \"be9e287a-187f-49b2-aafd-75205fcdff77\") " pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733110    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-var-lib-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-ovn\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733162    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-cni-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733188    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6bfc390-7582-4a98-b9f7-0b89c792034c-cni-binary-copy\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733212    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-sys\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733236    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-lib-modules\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733261    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-systemd-units\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733284    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-etc-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.733329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733307    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-system-cni-dir\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxr9\" (UniqueName: \"kubernetes.io/projected/52ec6eb9-7762-4821-ba44-87c584f1c5ac-kube-api-access-7zxr9\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733364    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-cni-multus\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmrk\" (UniqueName: \"kubernetes.io/projected/49ef1026-a078-4da8-aaf6-afca14956ed8-kube-api-access-2dmrk\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-systemd\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-cni-bin\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-sys-fs\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-kubernetes\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-systemd\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733657    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-conf-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-etc-kubernetes\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-slash\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.734069 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733812    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-cni-netd\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733869    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-run-ovn-kubernetes\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733885    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-kubelet\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733905    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysctl-d\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733938    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-tmp-dir\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.733975    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-socket-dir-parent\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-host\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734030    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssxp\" (UniqueName: \"kubernetes.io/projected/213d37c4-817c-4825-b29e-74e408822048-kube-api-access-nssxp\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734046    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-node-log\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734096    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734137    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.734794 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8f5n\" (UniqueName: \"kubernetes.io/projected/c048b030-3ada-4681-b3d7-3f177fa5122c-kube-api-access-k8f5n\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734209    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-tuned\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734227    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-modprobe-d\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovnkube-config\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-env-overrides\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734346    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcfq\" (UniqueName: \"kubernetes.io/projected/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-kube-api-access-nwcfq\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734379    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-os-release\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734414    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-run\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734436    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-k8s-cni-cncf-io\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734451    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5bl\" (UniqueName: \"kubernetes.io/projected/f6bfc390-7582-4a98-b9f7-0b89c792034c-kube-api-access-nx5bl\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49ef1026-a078-4da8-aaf6-afca14956ed8-iptables-alerter-script\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734485    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-var-lib-kubelet\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734520    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c642abb8-6e39-49db-955a-442707a3f0a6-kubelet-config\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734562    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c642abb8-6e39-49db-955a-442707a3f0a6-dbus\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.735457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-os-release\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-netns\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734664    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-kubelet\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be9e287a-187f-49b2-aafd-75205fcdff77-konnectivity-ca\") pod \"konnectivity-agent-zxldh\" (UID: \"be9e287a-187f-49b2-aafd-75205fcdff77\") " pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734724    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-run-netns\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734748    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovnkube-script-lib\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysconfig\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-cnibin\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734863    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-hostroot\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734888    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-socket-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-registration-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.734980    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9cb\" (UniqueName: \"kubernetes.io/projected/1e158eee-6883-41ae-996a-5dad5e07629b-kube-api-access-kf9cb\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.736223 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfmp\" (UniqueName: \"kubernetes.io/projected/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-kube-api-access-ncfmp\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.736770 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735092    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-system-cni-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.736770 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735137    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49ef1026-a078-4da8-aaf6-afca14956ed8-host-slash\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.736770 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735184    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-log-socket\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.736770 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-cni-bin\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.736770 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e158eee-6883-41ae-996a-5dad5e07629b-host\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.736770 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52ec6eb9-7762-4821-ba44-87c584f1c5ac-tmp\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.736770 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.735281    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-hosts-file\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.759835 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.759805    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zj7k"
Jul 17 19:56:07.772575 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.772553    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zj7k"
Jul 17 19:56:07.835604 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-sys-fs\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.835604 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-kubernetes\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.835604 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835590    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-systemd\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-conf-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-etc-kubernetes\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835666    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-systemd\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835691    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-kubernetes\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835718    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-slash\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835722    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-sys-fs\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835728    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-conf-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835757    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835756    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-etc-kubernetes\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835770    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-slash\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835742    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-cni-netd\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835811    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-cni-netd\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835844    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.835880 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-run-ovn-kubernetes\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835908    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-kubelet\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysctl-d\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.835983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-tmp-dir\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-socket-dir-parent\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-host\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836064    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssxp\" (UniqueName: \"kubernetes.io/projected/213d37c4-817c-4825-b29e-74e408822048-kube-api-access-nssxp\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836113    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-node-log\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836190    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8f5n\" (UniqueName: \"kubernetes.io/projected/c048b030-3ada-4681-b3d7-3f177fa5122c-kube-api-access-k8f5n\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836216    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-tuned\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-modprobe-d\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.836682 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836314    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836366    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-socket-dir-parent\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836386    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836490    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836578    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-node-log\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836593    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-host\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836567    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-kubelet\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836622    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836622    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836653    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-tmp-dir\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836665    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-modprobe-d\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836519    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysctl-d\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836651    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovnkube-config\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836719    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-env-overrides\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcfq\" (UniqueName: \"kubernetes.io/projected/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-kube-api-access-nwcfq\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.837377 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836627    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-run-ovn-kubernetes\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-os-release\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-run\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836885    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-k8s-cni-cncf-io\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836914    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5bl\" (UniqueName: \"kubernetes.io/projected/f6bfc390-7582-4a98-b9f7-0b89c792034c-kube-api-access-nx5bl\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836941    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49ef1026-a078-4da8-aaf6-afca14956ed8-iptables-alerter-script\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-var-lib-kubelet\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836998    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c642abb8-6e39-49db-955a-442707a3f0a6-kubelet-config\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.836999    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-run\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837023    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c642abb8-6e39-49db-955a-442707a3f0a6-dbus\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837047    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-os-release\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-netns\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837089    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-os-release\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-kubelet\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837156    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-k8s-cni-cncf-io\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838158 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837183    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c642abb8-6e39-49db-955a-442707a3f0a6-kubelet-config\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837202    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovnkube-config\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837200    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-os-release\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837223    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-env-overrides\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837271    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c642abb8-6e39-49db-955a-442707a3f0a6-dbus\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837248    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-netns\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837271    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-var-lib-kubelet\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837740    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be9e287a-187f-49b2-aafd-75205fcdff77-konnectivity-ca\") pod \"konnectivity-agent-zxldh\" (UID: \"be9e287a-187f-49b2-aafd-75205fcdff77\") " pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837775    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-run-netns\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovnkube-script-lib\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49ef1026-a078-4da8-aaf6-afca14956ed8-iptables-alerter-script\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837858    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-kubelet\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837907    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837937    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysconfig\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-cnibin\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.838887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837978    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-hostroot\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.837997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-socket-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838029    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-registration-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9cb\" (UniqueName: \"kubernetes.io/projected/1e158eee-6883-41ae-996a-5dad5e07629b-kube-api-access-kf9cb\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838081    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysconfig\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfmp\" (UniqueName: \"kubernetes.io/projected/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-kube-api-access-ncfmp\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838146    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-system-cni-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49ef1026-a078-4da8-aaf6-afca14956ed8-host-slash\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838192    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-log-socket\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-cni-bin\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e158eee-6883-41ae-996a-5dad5e07629b-host\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52ec6eb9-7762-4821-ba44-87c584f1c5ac-tmp\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838273    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-hosts-file\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.839645 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838297    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-device-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysctl-conf\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838324    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovnkube-script-lib\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838346    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-daemon-config\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838357    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be9e287a-187f-49b2-aafd-75205fcdff77-konnectivity-ca\") pod \"konnectivity-agent-zxldh\" (UID: \"be9e287a-187f-49b2-aafd-75205fcdff77\") " pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838382    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-run-netns\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838453    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-system-cni-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838459    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838493    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovn-node-metrics-cert\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-cnibin\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838547    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e158eee-6883-41ae-996a-5dad5e07629b-serviceca\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838573    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838601    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-multus-certs\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.840348 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9hm\" (UniqueName: \"kubernetes.io/projected/bf6b5042-e40a-4549-a761-2ef68ad8c71b-kube-api-access-qw9hm\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838648    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-hosts-file\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be9e287a-187f-49b2-aafd-75205fcdff77-agent-certs\") pod \"konnectivity-agent-zxldh\" (UID: \"be9e287a-187f-49b2-aafd-75205fcdff77\") " pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-var-lib-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838715    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-ovn\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-cni-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6bfc390-7582-4a98-b9f7-0b89c792034c-cni-binary-copy\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838783    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-sysctl-conf\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-sys\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838827    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-lib-modules\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838850    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-systemd-units\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838876    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-etc-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838901    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-system-cni-dir\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxr9\" (UniqueName: \"kubernetes.io/projected/52ec6eb9-7762-4821-ba44-87c584f1c5ac-kube-api-access-7zxr9\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838954    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-cni-multus\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.841087 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmrk\" (UniqueName: \"kubernetes.io/projected/49ef1026-a078-4da8-aaf6-afca14956ed8-kube-api-access-2dmrk\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-systemd\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839033    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-cni-bin\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839061    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.839201    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839212    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49ef1026-a078-4da8-aaf6-afca14956ed8-host-slash\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839274    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-log-socket\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.839286    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs podName:bf6b5042-e40a-4549-a761-2ef68ad8c71b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:08.339248652 +0000 UTC m=+3.033224804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs") pod "network-metrics-daemon-b5zcw" (UID: "bf6b5042-e40a-4549-a761-2ef68ad8c71b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839330    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e158eee-6883-41ae-996a-5dad5e07629b-host\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839343    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-device-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839348    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-cnibin\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839376    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-etc-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839420    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-cni-multus\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839425    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-var-lib-cni-bin\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839378    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-host-cni-bin\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.841882 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839408    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-hostroot\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839485    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-systemd\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839650    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-var-lib-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.838495    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-openvswitch\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839650    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-cni-dir\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839376    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6bfc390-7582-4a98-b9f7-0b89c792034c-multus-daemon-config\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.839733    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.839787    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret podName:c642abb8-6e39-49db-955a-442707a3f0a6 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:08.339767705 +0000 UTC m=+3.033743831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret") pod "global-pull-secret-syncer-mdlgw" (UID: "c642abb8-6e39-49db-955a-442707a3f0a6") : object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839831    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6bfc390-7582-4a98-b9f7-0b89c792034c-host-run-multus-certs\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839882    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-systemd-units\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839930    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-run-ovn\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.839985    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-socket-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.840007    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.842626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.840030    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-system-cni-dir\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.845042 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.845012    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-sys\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.845170 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.845150    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52ec6eb9-7762-4821-ba44-87c584f1c5ac-lib-modules\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.845213 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.845201    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c048b030-3ada-4681-b3d7-3f177fa5122c-cnibin\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.845547 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.845524    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e158eee-6883-41ae-996a-5dad5e07629b-serviceca\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.845887 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.845865    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c048b030-3ada-4681-b3d7-3f177fa5122c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.846499 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.846234    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-ovn-node-metrics-cert\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.846652 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.846590    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6bfc390-7582-4a98-b9f7-0b89c792034c-cni-binary-copy\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.847022 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.847001    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be9e287a-187f-49b2-aafd-75205fcdff77-agent-certs\") pod \"konnectivity-agent-zxldh\" (UID: \"be9e287a-187f-49b2-aafd-75205fcdff77\") " pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:07.847237 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.847218    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52ec6eb9-7762-4821-ba44-87c584f1c5ac-tmp\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.849112 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.849020    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 19:56:07.849112 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.849044    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 19:56:07.849112 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.849101    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fb5qf for pod openshift-network-diagnostics/network-check-target-stgxg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:07.849322 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:07.849172    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf podName:5ddde34f-c002-4405-a98d-ce52d47a9bdb nodeName:}" failed. No retries permitted until 2026-07-17 19:56:08.349144363 +0000 UTC m=+3.043120508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fb5qf" (UniqueName: "kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf") pod "network-check-target-stgxg" (UID: "5ddde34f-c002-4405-a98d-ce52d47a9bdb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:07.850196 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.850151    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/213d37c4-817c-4825-b29e-74e408822048-registration-dir\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.850970 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.850944    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5bl\" (UniqueName: \"kubernetes.io/projected/f6bfc390-7582-4a98-b9f7-0b89c792034c-kube-api-access-nx5bl\") pod \"multus-vjss5\" (UID: \"f6bfc390-7582-4a98-b9f7-0b89c792034c\") " pod="openshift-multus/multus-vjss5"
Jul 17 19:56:07.854816 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.854731    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmrk\" (UniqueName: \"kubernetes.io/projected/49ef1026-a078-4da8-aaf6-afca14956ed8-kube-api-access-2dmrk\") pod \"iptables-alerter-ghpwh\" (UID: \"49ef1026-a078-4da8-aaf6-afca14956ed8\") " pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:07.854914 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.854811    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssxp\" (UniqueName: \"kubernetes.io/projected/213d37c4-817c-4825-b29e-74e408822048-kube-api-access-nssxp\") pod \"aws-ebs-csi-driver-node-5ng9t\" (UID: \"213d37c4-817c-4825-b29e-74e408822048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:07.856167 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.855957    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8f5n\" (UniqueName: \"kubernetes.io/projected/c048b030-3ada-4681-b3d7-3f177fa5122c-kube-api-access-k8f5n\") pod \"multus-additional-cni-plugins-jt22m\" (UID: \"c048b030-3ada-4681-b3d7-3f177fa5122c\") " pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:07.856167 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.856133    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcfq\" (UniqueName: \"kubernetes.io/projected/c6fe3fbc-3b7d-4df5-9a09-12d85016fa68-kube-api-access-nwcfq\") pod \"ovnkube-node-9tgrl\" (UID: \"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68\") " pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:07.857020 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.856987    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal" event={"ID":"fee423c54644a9085fb43b1dbbd566c9","Type":"ContainerStarted","Data":"c54d65f1195186da0883e5449c0d2e7cd158a8ee5c1be9b41645a71dd11c04aa"}
Jul 17 19:56:07.857284 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.857261    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52ec6eb9-7762-4821-ba44-87c584f1c5ac-etc-tuned\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.857764 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.857739    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9hm\" (UniqueName: \"kubernetes.io/projected/bf6b5042-e40a-4549-a761-2ef68ad8c71b-kube-api-access-qw9hm\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:07.858248 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.858225    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfmp\" (UniqueName: \"kubernetes.io/projected/031d2f1f-e63a-4d02-ac4e-bd7a4eca7773-kube-api-access-ncfmp\") pod \"node-resolver-5z8z9\" (UID: \"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773\") " pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.858341 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.858230    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxr9\" (UniqueName: \"kubernetes.io/projected/52ec6eb9-7762-4821-ba44-87c584f1c5ac-kube-api-access-7zxr9\") pod \"tuned-58hlf\" (UID: \"52ec6eb9-7762-4821-ba44-87c584f1c5ac\") " pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.867076 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.867050    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9cb\" (UniqueName: \"kubernetes.io/projected/1e158eee-6883-41ae-996a-5dad5e07629b-kube-api-access-kf9cb\") pod \"node-ca-pjbwc\" (UID: \"1e158eee-6883-41ae-996a-5dad5e07629b\") " pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:07.980322 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.980283    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-58hlf"
Jul 17 19:56:07.988328 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.988301    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5z8z9"
Jul 17 19:56:07.998234 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:07.998212    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:08.005161 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.005141    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjss5"
Jul 17 19:56:08.013149 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.013129    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t"
Jul 17 19:56:08.020985 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.020962    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:08.037701 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.037677    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jt22m"
Jul 17 19:56:08.046698 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.046677    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghpwh"
Jul 17 19:56:08.053378 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.053355    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pjbwc"
Jul 17 19:56:08.342489 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.342462    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:08.342689 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.342536    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:08.342689 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.342626    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:08.342689 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.342637    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:08.342817 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.342695    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret podName:c642abb8-6e39-49db-955a-442707a3f0a6 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:09.342677566 +0000 UTC m=+4.036653712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret") pod "global-pull-secret-syncer-mdlgw" (UID: "c642abb8-6e39-49db-955a-442707a3f0a6") : object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:08.342817 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.342712    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs podName:bf6b5042-e40a-4549-a761-2ef68ad8c71b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:09.342704599 +0000 UTC m=+4.036680738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs") pod "network-metrics-daemon-b5zcw" (UID: "bf6b5042-e40a-4549-a761-2ef68ad8c71b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:08.443132 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.443085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:08.443285 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.443223    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 19:56:08.443285 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.443246    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 19:56:08.443285 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.443257    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fb5qf for pod openshift-network-diagnostics/network-check-target-stgxg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:08.443422 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:08.443310    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf podName:5ddde34f-c002-4405-a98d-ce52d47a9bdb nodeName:}" failed. No retries permitted until 2026-07-17 19:56:09.443296311 +0000 UTC m=+4.137272436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb5qf" (UniqueName: "kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf") pod "network-check-target-stgxg" (UID: "5ddde34f-c002-4405-a98d-ce52d47a9bdb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:08.775515 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.775434    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 19:51:07 +0000 UTC" deadline="2027-12-31 09:14:37.294427562 +0000 UTC"
Jul 17 19:56:08.775515 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.775466    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12757h18m28.518965665s"
Jul 17 19:56:08.825847 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.825815    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9e287a_187f_49b2_aafd_75205fcdff77.slice/crio-ff2db505ab0dfdac7a70ac8ab4af4aaaddda9441fc0485415896dc6c331d5736 WatchSource:0}: Error finding container ff2db505ab0dfdac7a70ac8ab4af4aaaddda9441fc0485415896dc6c331d5736: Status 404 returned error can't find the container with id ff2db505ab0dfdac7a70ac8ab4af4aaaddda9441fc0485415896dc6c331d5736
Jul 17 19:56:08.828271 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.827981    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fe3fbc_3b7d_4df5_9a09_12d85016fa68.slice/crio-aa769a87927716506ce24407b3dd2f13f2b1c8cd3c576918507946c409d6d00b WatchSource:0}: Error finding container aa769a87927716506ce24407b3dd2f13f2b1c8cd3c576918507946c409d6d00b: Status 404 returned error can't find the container with id aa769a87927716506ce24407b3dd2f13f2b1c8cd3c576918507946c409d6d00b
Jul 17 19:56:08.830583 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.830425    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bfc390_7582_4a98_b9f7_0b89c792034c.slice/crio-0bcdb09501430eb9af0baaaf4f7f8d7c91ae3a32c51ddb2e183f6fb631476466 WatchSource:0}: Error finding container 0bcdb09501430eb9af0baaaf4f7f8d7c91ae3a32c51ddb2e183f6fb631476466: Status 404 returned error can't find the container with id 0bcdb09501430eb9af0baaaf4f7f8d7c91ae3a32c51ddb2e183f6fb631476466
Jul 17 19:56:08.834724 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.834688    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ef1026_a078_4da8_aaf6_afca14956ed8.slice/crio-5568a634580a2fb8129c6b6b39b7fbfe8859b94ac5653f0692fb4194fe84fbd7 WatchSource:0}: Error finding container 5568a634580a2fb8129c6b6b39b7fbfe8859b94ac5653f0692fb4194fe84fbd7: Status 404 returned error can't find the container with id 5568a634580a2fb8129c6b6b39b7fbfe8859b94ac5653f0692fb4194fe84fbd7
Jul 17 19:56:08.835554 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.835517    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213d37c4_817c_4825_b29e_74e408822048.slice/crio-813fc6ecaa9da50b3428625969b216020856335ba1f0463d7424b625bdeb249d WatchSource:0}: Error finding container 813fc6ecaa9da50b3428625969b216020856335ba1f0463d7424b625bdeb249d: Status 404 returned error can't find the container with id 813fc6ecaa9da50b3428625969b216020856335ba1f0463d7424b625bdeb249d
Jul 17 19:56:08.836398 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.836276    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ec6eb9_7762_4821_ba44_87c584f1c5ac.slice/crio-4938b5b368fc65c06122613d7ec6980ebe9f4ed2f029355a94727d7e9df8a2bc WatchSource:0}: Error finding container 4938b5b368fc65c06122613d7ec6980ebe9f4ed2f029355a94727d7e9df8a2bc: Status 404 returned error can't find the container with id 4938b5b368fc65c06122613d7ec6980ebe9f4ed2f029355a94727d7e9df8a2bc
Jul 17 19:56:08.837951 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.837873    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031d2f1f_e63a_4d02_ac4e_bd7a4eca7773.slice/crio-54e42f30922df89d3ff478966bec4ac4c482eddad4c8646690c0bbd466aa8f57 WatchSource:0}: Error finding container 54e42f30922df89d3ff478966bec4ac4c482eddad4c8646690c0bbd466aa8f57: Status 404 returned error can't find the container with id 54e42f30922df89d3ff478966bec4ac4c482eddad4c8646690c0bbd466aa8f57
Jul 17 19:56:08.840773 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.839865    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc048b030_3ada_4681_b3d7_3f177fa5122c.slice/crio-978186babcb753af35111eb7b2adf06be13b71edbdc52b41b8840b16cd431a5a WatchSource:0}: Error finding container 978186babcb753af35111eb7b2adf06be13b71edbdc52b41b8840b16cd431a5a: Status 404 returned error can't find the container with id 978186babcb753af35111eb7b2adf06be13b71edbdc52b41b8840b16cd431a5a
Jul 17 19:56:08.840773 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:08.840321    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e158eee_6883_41ae_996a_5dad5e07629b.slice/crio-fda50642e9f0f7d527eb48af97c01c0bc9af0e8e4cae2209d4d9d3c113058de1 WatchSource:0}: Error finding container fda50642e9f0f7d527eb48af97c01c0bc9af0e8e4cae2209d4d9d3c113058de1: Status 404 returned error can't find the container with id fda50642e9f0f7d527eb48af97c01c0bc9af0e8e4cae2209d4d9d3c113058de1
Jul 17 19:56:08.860826 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.860795    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pjbwc" event={"ID":"1e158eee-6883-41ae-996a-5dad5e07629b","Type":"ContainerStarted","Data":"fda50642e9f0f7d527eb48af97c01c0bc9af0e8e4cae2209d4d9d3c113058de1"}
Jul 17 19:56:08.864837 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.864810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerStarted","Data":"978186babcb753af35111eb7b2adf06be13b71edbdc52b41b8840b16cd431a5a"}
Jul 17 19:56:08.867713 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.867688    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5z8z9" event={"ID":"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773","Type":"ContainerStarted","Data":"54e42f30922df89d3ff478966bec4ac4c482eddad4c8646690c0bbd466aa8f57"}
Jul 17 19:56:08.872594 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.872568    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-58hlf" event={"ID":"52ec6eb9-7762-4821-ba44-87c584f1c5ac","Type":"ContainerStarted","Data":"4938b5b368fc65c06122613d7ec6980ebe9f4ed2f029355a94727d7e9df8a2bc"}
Jul 17 19:56:08.873737 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.873705    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t" event={"ID":"213d37c4-817c-4825-b29e-74e408822048","Type":"ContainerStarted","Data":"813fc6ecaa9da50b3428625969b216020856335ba1f0463d7424b625bdeb249d"}
Jul 17 19:56:08.875210 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.875189    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"aa769a87927716506ce24407b3dd2f13f2b1c8cd3c576918507946c409d6d00b"}
Jul 17 19:56:08.876185 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.876156    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghpwh" event={"ID":"49ef1026-a078-4da8-aaf6-afca14956ed8","Type":"ContainerStarted","Data":"5568a634580a2fb8129c6b6b39b7fbfe8859b94ac5653f0692fb4194fe84fbd7"}
Jul 17 19:56:08.877274 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.877256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjss5" event={"ID":"f6bfc390-7582-4a98-b9f7-0b89c792034c","Type":"ContainerStarted","Data":"0bcdb09501430eb9af0baaaf4f7f8d7c91ae3a32c51ddb2e183f6fb631476466"}
Jul 17 19:56:08.878196 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:08.878173    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxldh" event={"ID":"be9e287a-187f-49b2-aafd-75205fcdff77","Type":"ContainerStarted","Data":"ff2db505ab0dfdac7a70ac8ab4af4aaaddda9441fc0485415896dc6c331d5736"}
Jul 17 19:56:09.349279 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.348975    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:09.349279 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.349274    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:09.349509 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.349413    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:09.349509 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.349473    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs podName:bf6b5042-e40a-4549-a761-2ef68ad8c71b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:11.349455823 +0000 UTC m=+6.043431947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs") pod "network-metrics-daemon-b5zcw" (UID: "bf6b5042-e40a-4549-a761-2ef68ad8c71b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:09.349622 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.349536    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:09.349622 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.349566    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret podName:c642abb8-6e39-49db-955a-442707a3f0a6 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:11.349556121 +0000 UTC m=+6.043532259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret") pod "global-pull-secret-syncer-mdlgw" (UID: "c642abb8-6e39-49db-955a-442707a3f0a6") : object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:09.453381 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.453336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:09.453546 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.453505    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 19:56:09.453546 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.453526    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 19:56:09.453546 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.453538    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fb5qf for pod openshift-network-diagnostics/network-check-target-stgxg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:09.453686 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.453596    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf podName:5ddde34f-c002-4405-a98d-ce52d47a9bdb nodeName:}" failed. No retries permitted until 2026-07-17 19:56:11.453575293 +0000 UTC m=+6.147551417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb5qf" (UniqueName: "kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf") pod "network-check-target-stgxg" (UID: "5ddde34f-c002-4405-a98d-ce52d47a9bdb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:09.776705 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.776594    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 19:51:07 +0000 UTC" deadline="2028-03-15 09:44:33.456446479 +0000 UTC"
Jul 17 19:56:09.776705 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.776630    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14557h48m23.679821126s"
Jul 17 19:56:09.859208 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.858189    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:09.859208 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.858376    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:09.859208 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.858866    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:09.859208 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.858988    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:09.866656 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.866483    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:09.866656 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:09.866614    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:09.898148 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.898095    2572 generic.go:356] "Generic (PLEG): container finished" podID="6594032f970e6bed2e3d6e0abc512be7" containerID="e98202a3cd73b09ab59500475e149ae6207113233c1ec31fa858148b1d8ecf38" exitCode=0
Jul 17 19:56:09.898319 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.898183    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal" event={"ID":"6594032f970e6bed2e3d6e0abc512be7","Type":"ContainerDied","Data":"e98202a3cd73b09ab59500475e149ae6207113233c1ec31fa858148b1d8ecf38"}
Jul 17 19:56:09.911915 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.911880    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal" event={"ID":"fee423c54644a9085fb43b1dbbd566c9","Type":"ContainerStarted","Data":"ccd09f68aa5afa3178a1419a2198a06a0d27018d605f7e2a578dfc58ed5d1536"}
Jul 17 19:56:09.939335 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:09.939274    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-185.ec2.internal" podStartSLOduration=2.939253023 podStartE2EDuration="2.939253023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:09.938844907 +0000 UTC m=+4.632821056" watchObservedRunningTime="2026-07-17 19:56:09.939253023 +0000 UTC m=+4.633229170"
Jul 17 19:56:10.942343 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:10.942121    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal" event={"ID":"6594032f970e6bed2e3d6e0abc512be7","Type":"ContainerStarted","Data":"4529ca96ecb7b4a407f2b08cee73b2d4ff0a856a05447204c62f254254114e04"}
Jul 17 19:56:11.372410 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:11.372328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:11.372410 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:11.372383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:11.372709 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.372515    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:11.372709 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.372574    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs podName:bf6b5042-e40a-4549-a761-2ef68ad8c71b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:15.37255644 +0000 UTC m=+10.066532577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs") pod "network-metrics-daemon-b5zcw" (UID: "bf6b5042-e40a-4549-a761-2ef68ad8c71b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:11.372960 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.372940    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:11.373090 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.373001    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret podName:c642abb8-6e39-49db-955a-442707a3f0a6 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:15.372985544 +0000 UTC m=+10.066961673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret") pod "global-pull-secret-syncer-mdlgw" (UID: "c642abb8-6e39-49db-955a-442707a3f0a6") : object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:11.472812 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:11.472757    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:11.472984 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.472936    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 19:56:11.472984 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.472958    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 19:56:11.472984 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.472971    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fb5qf for pod openshift-network-diagnostics/network-check-target-stgxg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:11.473170 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.473023    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf podName:5ddde34f-c002-4405-a98d-ce52d47a9bdb nodeName:}" failed. No retries permitted until 2026-07-17 19:56:15.473004629 +0000 UTC m=+10.166980772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb5qf" (UniqueName: "kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf") pod "network-check-target-stgxg" (UID: "5ddde34f-c002-4405-a98d-ce52d47a9bdb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:11.850236 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:11.850161    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:11.850395 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:11.850262    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:11.850395 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.850380    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:11.850822 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:11.850610    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:11.850822 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.850729    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:11.851359 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:11.851203    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:13.848077 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:13.847986    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:13.848077 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:13.848017    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:13.848077 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:13.848059    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:13.850698 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:13.849279    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:13.850698 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:13.849447    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:13.850698 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:13.849577    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:15.408083 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:15.408030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:15.408498 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:15.408105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:15.408498 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.408239    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:15.408498 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.408300    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs podName:bf6b5042-e40a-4549-a761-2ef68ad8c71b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:23.408281094 +0000 UTC m=+18.102257232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs") pod "network-metrics-daemon-b5zcw" (UID: "bf6b5042-e40a-4549-a761-2ef68ad8c71b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:15.408498 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.408362    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:15.408498 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.408392    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret podName:c642abb8-6e39-49db-955a-442707a3f0a6 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:23.408381849 +0000 UTC m=+18.102357977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret") pod "global-pull-secret-syncer-mdlgw" (UID: "c642abb8-6e39-49db-955a-442707a3f0a6") : object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:15.509191 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:15.509153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:15.509361 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.509306    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 19:56:15.509361 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.509331    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 19:56:15.509361 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.509344    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fb5qf for pod openshift-network-diagnostics/network-check-target-stgxg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:15.509517 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.509409    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf podName:5ddde34f-c002-4405-a98d-ce52d47a9bdb nodeName:}" failed. No retries permitted until 2026-07-17 19:56:23.509388948 +0000 UTC m=+18.203365077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb5qf" (UniqueName: "kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf") pod "network-check-target-stgxg" (UID: "5ddde34f-c002-4405-a98d-ce52d47a9bdb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:15.850097 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:15.849342    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:15.850097 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.849459    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:15.850097 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:15.849531    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:15.850097 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.849614    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:15.850097 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:15.849956    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:15.850097 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:15.850038    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:17.848415 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:17.848382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:17.848415 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:17.848415    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:17.848873 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:17.848383    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:17.848873 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:17.848507    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:17.848873 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:17.848610    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:17.848873 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:17.848698    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:19.848292 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:19.848261    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:19.848727 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:19.848261    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:19.848727 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:19.848387    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:19.848727 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:19.848467    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:19.848727 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:19.848266    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:19.848727 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:19.848568    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:21.848287 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:21.848252    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:21.848837 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:21.848258    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:21.848837 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:21.848399    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:21.848837 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:21.848264    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:21.848837 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:21.848474    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:21.848837 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:21.848532    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:23.467080 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:23.467019    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:23.467600 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:23.467099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:23.467600 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.467149    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:23.467600 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.467194    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:23.467600 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.467226    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret podName:c642abb8-6e39-49db-955a-442707a3f0a6 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:39.46720511 +0000 UTC m=+34.161181236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret") pod "global-pull-secret-syncer-mdlgw" (UID: "c642abb8-6e39-49db-955a-442707a3f0a6") : object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:23.467600 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.467242    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs podName:bf6b5042-e40a-4549-a761-2ef68ad8c71b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:39.467234992 +0000 UTC m=+34.161211116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs") pod "network-metrics-daemon-b5zcw" (UID: "bf6b5042-e40a-4549-a761-2ef68ad8c71b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:23.567736 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:23.567700    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:23.567921 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.567898    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 19:56:23.567959 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.567928    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 19:56:23.567959 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.567945    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fb5qf for pod openshift-network-diagnostics/network-check-target-stgxg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:23.568083 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.568016    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf podName:5ddde34f-c002-4405-a98d-ce52d47a9bdb nodeName:}" failed. No retries permitted until 2026-07-17 19:56:39.567997315 +0000 UTC m=+34.261973445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb5qf" (UniqueName: "kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf") pod "network-check-target-stgxg" (UID: "5ddde34f-c002-4405-a98d-ce52d47a9bdb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:23.847740 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:23.847657    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:23.847901 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:23.847662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:23.847901 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.847787    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:23.847901 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.847869    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:23.847901 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:23.847676    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:23.848117 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:23.847954    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:25.848688 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:25.848653    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:25.849161 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:25.848766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:25.849161 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:25.848838    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:25.849161 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:25.848931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:25.849161 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:25.848990    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:25.849161 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:25.848999    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:27.850033 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.849825    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:27.850796 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.849825    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:27.850796 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:27.850153    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:27.850796 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:27.850262    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:27.850796 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.849825    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:27.850796 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:27.850371    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:27.978262 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.978229    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pjbwc" event={"ID":"1e158eee-6883-41ae-996a-5dad5e07629b","Type":"ContainerStarted","Data":"ab1a4fc16a2fcaeba5b8fe127be50ae32904f0c9b0a9d6c4076c367c1e122562"}
Jul 17 19:56:27.979782 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.979758    2572 generic.go:356] "Generic (PLEG): container finished" podID="c048b030-3ada-4681-b3d7-3f177fa5122c" containerID="6268b819b50f34748f97c00170a8674df0115f203c0f268704fa562b8ee59c56" exitCode=0
Jul 17 19:56:27.979884 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.979835    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerDied","Data":"6268b819b50f34748f97c00170a8674df0115f203c0f268704fa562b8ee59c56"}
Jul 17 19:56:27.981167 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.981149    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5z8z9" event={"ID":"031d2f1f-e63a-4d02-ac4e-bd7a4eca7773","Type":"ContainerStarted","Data":"41e68dfc5dfbfec449bd8ce9d894c50c677573ac2403e363b8d26fc81bfcf083"}
Jul 17 19:56:27.982581 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.982560    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-58hlf" event={"ID":"52ec6eb9-7762-4821-ba44-87c584f1c5ac","Type":"ContainerStarted","Data":"0ede7e492c918426b3e6978ddbf6388c3229d8c128691f7cc610b3ec4f9c8e3d"}
Jul 17 19:56:27.983949 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.983932    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t" event={"ID":"213d37c4-817c-4825-b29e-74e408822048","Type":"ContainerStarted","Data":"580db6fcdd5495f34cc36503a5431744715797702814d164f46b3064470982ad"}
Jul 17 19:56:27.985802 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.985784    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"83ccd9161ac562e7428728789f9616ccd9a948c080a99816e20e2d46c3c10222"}
Jul 17 19:56:27.985889 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.985859    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"ea0d967c41e934ad4f04726678a3a819dca5a4d02007ce67c73af2fce64543fd"}
Jul 17 19:56:27.985930 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.985906    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"d1a411d37eca68d5e9d9f761379552a718aa9c3c2c739ae2d208b2f30ed25287"}
Jul 17 19:56:27.987164 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.987147    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjss5" event={"ID":"f6bfc390-7582-4a98-b9f7-0b89c792034c","Type":"ContainerStarted","Data":"9ee3009ab8ad9130a1bf6b6d6f014b1c73264ed39215a47a190c488167c9f0c5"}
Jul 17 19:56:27.988372 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.988356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxldh" event={"ID":"be9e287a-187f-49b2-aafd-75205fcdff77","Type":"ContainerStarted","Data":"1e96640e76662986664e791957b8b05c0edb1f1fb13e1cab786c3ed3e66ed89d"}
Jul 17 19:56:27.999036 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:27.998999    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-185.ec2.internal" podStartSLOduration=20.998971044 podStartE2EDuration="20.998971044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:10.958340118 +0000 UTC m=+5.652316265" watchObservedRunningTime="2026-07-17 19:56:27.998971044 +0000 UTC m=+22.692947191"
Jul 17 19:56:28.044182 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.044142    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pjbwc" podStartSLOduration=3.4344707899999998 podStartE2EDuration="22.044126987s" totalImagesPullingTime="18.609656197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:28.002173791 +0000 UTC m=+22.696149936" watchObservedRunningTime="2026-07-17 19:56:28.044126987 +0000 UTC m=+22.738103133"
Jul 17 19:56:28.066526 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.066491    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5z8z9" podStartSLOduration=8.176977301 podStartE2EDuration="22.066479368s" totalImagesPullingTime="13.889502067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:28.066224102 +0000 UTC m=+22.760200249" watchObservedRunningTime="2026-07-17 19:56:28.066479368 +0000 UTC m=+22.760455513"
Jul 17 19:56:28.088573 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.088538    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zxldh" podStartSLOduration=11.873423017 podStartE2EDuration="22.088523393s" totalImagesPullingTime="10.215100376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:28.087918277 +0000 UTC m=+22.781894423" watchObservedRunningTime="2026-07-17 19:56:28.088523393 +0000 UTC m=+22.782499539"
Jul 17 19:56:28.111078 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.110998    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vjss5" podStartSLOduration=3.45274557 podStartE2EDuration="22.110985254s" totalImagesPullingTime="18.658239684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:28.110695451 +0000 UTC m=+22.804671600" watchObservedRunningTime="2026-07-17 19:56:28.110985254 +0000 UTC m=+22.804961401"
Jul 17 19:56:28.132271 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.132230    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-58hlf" podStartSLOduration=3.501432051 podStartE2EDuration="22.132215239s" totalImagesPullingTime="18.630783188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:28.131604999 +0000 UTC m=+22.825581144" watchObservedRunningTime="2026-07-17 19:56:28.132215239 +0000 UTC m=+22.826191384"
Jul 17 19:56:28.948510 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.948482    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 19:56:28.993080 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.993032    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t" event={"ID":"213d37c4-817c-4825-b29e-74e408822048","Type":"ContainerStarted","Data":"4654c6d4d5da1d572144883ba78d95f3353df7aae57c93e6e3b89c7becb99ac4"}
Jul 17 19:56:28.996499 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.996474    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"ab1cfcaaf60ddcd19c00521c345e6843ff21c09126680e512559cf191440975e"}
Jul 17 19:56:28.996612 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.996556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"14541b44463fa30dc7b3b212c03bd9a727e649154d92463800fde2a7f40b23a4"}
Jul 17 19:56:28.996612 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.996600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"bd21874a29ad60b74e718662ebc299dcbe5dc4a772bd833740eb114d91ec12e1"}
Jul 17 19:56:28.998082 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:28.998002    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghpwh" event={"ID":"49ef1026-a078-4da8-aaf6-afca14956ed8","Type":"ContainerStarted","Data":"09cdfd2f11442027112c7fcdbb59767b949603e1bf44d708cade18505685bf19"}
Jul 17 19:56:29.395889 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.395858    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:29.396996 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.396574    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:29.421750 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.421685    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ghpwh" podStartSLOduration=4.789843419 podStartE2EDuration="23.421670478s" totalImagesPullingTime="18.631827059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:29.022893638 +0000 UTC m=+23.716869845" watchObservedRunningTime="2026-07-17 19:56:29.421670478 +0000 UTC m=+24.115646623"
Jul 17 19:56:29.790506 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.790401    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T19:56:28.948505173Z","UUID":"009affba-b68f-4c8a-96c9-cf0e6cf4a48e","Handler":null,"Name":"","Endpoint":""}
Jul 17 19:56:29.791983 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.791960    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 19:56:29.792122 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.792003    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 19:56:29.848293 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.848265    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:29.848469 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.848278    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:29.848469 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:29.848395    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:29.848581 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:29.848463    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:29.848581 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:29.848567    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:29.848699 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:29.848678    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:31.005327 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.005286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t" event={"ID":"213d37c4-817c-4825-b29e-74e408822048","Type":"ContainerStarted","Data":"737f964fa9a35c1b139f7888742dc809e7fc6878e43d30c7f9dec7cb05d1565e"}
Jul 17 19:56:31.009035 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.009008    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 17 19:56:31.009035 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.009019    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"a43639b965f7171fb0eed1474bfb3e6da27586085b2200e1f52c98df63332ab0"}
Jul 17 19:56:31.027363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.027314    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ng9t" podStartSLOduration=4.276145516 podStartE2EDuration="25.02729663s" totalImagesPullingTime="20.751151114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:31.027228317 +0000 UTC m=+25.721204474" watchObservedRunningTime="2026-07-17 19:56:31.02729663 +0000 UTC m=+25.721272779"
Jul 17 19:56:31.690275 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.690237    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:31.690997 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.690971    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zxldh"
Jul 17 19:56:31.848335 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.848298    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:31.848497 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.848304    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:31.848497 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:31.848415    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:31.848497 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:31.848310    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:31.848654 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:31.848495    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:31.848654 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:31.848573    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:33.017950 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.017741    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" event={"ID":"c6fe3fbc-3b7d-4df5-9a09-12d85016fa68","Type":"ContainerStarted","Data":"f096571d31e45ccb09eb82820b793ef42883756d31164149d5d79759c3309e74"}
Jul 17 19:56:33.018793 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.017986    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:33.018793 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.018032    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:33.019671 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.019645    2572 generic.go:356] "Generic (PLEG): container finished" podID="c048b030-3ada-4681-b3d7-3f177fa5122c" containerID="fb4b26d968d7b4d50f23a73b342a74d620ee9628b3b712a732f384842bb8cc8d" exitCode=0
Jul 17 19:56:33.019785 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.019676    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerDied","Data":"fb4b26d968d7b4d50f23a73b342a74d620ee9628b3b712a732f384842bb8cc8d"}
Jul 17 19:56:33.033664 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.033643    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:33.098761 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.098704    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl" podStartSLOduration=8.390670010000001 podStartE2EDuration="27.09869066s" totalImagesPullingTime="18.70802065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:33.06608483 +0000 UTC m=+27.760060977" watchObservedRunningTime="2026-07-17 19:56:33.09869066 +0000 UTC m=+27.792666806"
Jul 17 19:56:33.851977 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.849973    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:33.851977 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:33.850159    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:33.851977 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.850427    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:33.851977 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:33.850538    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:33.851977 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:33.850680    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:33.851977 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:33.850777    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:34.022808 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.022602    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:34.036927 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.036905    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:56:34.786065 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.786025    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b5zcw"]
Jul 17 19:56:34.786225 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.786134    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:34.786225 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:34.786218    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:34.792864 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.792802    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mdlgw"]
Jul 17 19:56:34.792990 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.792894    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:34.793042 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:34.792989    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:34.795456 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.795418    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stgxg"]
Jul 17 19:56:34.795589 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:34.795500    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:34.795659 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:34.795588    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:35.026697 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:35.026663    2572 generic.go:356] "Generic (PLEG): container finished" podID="c048b030-3ada-4681-b3d7-3f177fa5122c" containerID="a104a3d76356dc9ce9ddeb82647bc86860ff53935b480df9eaef8f172e8570b0" exitCode=0
Jul 17 19:56:35.027225 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:35.026749    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerDied","Data":"a104a3d76356dc9ce9ddeb82647bc86860ff53935b480df9eaef8f172e8570b0"}
Jul 17 19:56:35.848718 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:35.848688    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:35.849031 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:35.848759    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:35.851583 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:35.849141    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:35.851583 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:35.849151    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:35.851583 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:35.849267    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:35.851583 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:35.849411    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:37.034215 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:37.034182    2572 generic.go:356] "Generic (PLEG): container finished" podID="c048b030-3ada-4681-b3d7-3f177fa5122c" containerID="f9eb837ef7b4417828271afc526a0314813b694c248a758dfced7d1d5f687dbf" exitCode=0
Jul 17 19:56:37.034569 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:37.034244    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerDied","Data":"f9eb837ef7b4417828271afc526a0314813b694c248a758dfced7d1d5f687dbf"}
Jul 17 19:56:37.848373 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:37.848132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:37.848539 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:37.848131    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:37.848539 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:37.848479    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:37.848539 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:37.848137    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:37.848678 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:37.848571    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:37.848678 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:37.848619    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:39.486175 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:39.486138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:39.486708 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:39.486198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:39.486708 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.486359    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:39.486708 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.486390    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:39.486708 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.486655    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs podName:bf6b5042-e40a-4549-a761-2ef68ad8c71b nodeName:}" failed. No retries permitted until 2026-07-17 19:57:11.486426792 +0000 UTC m=+66.180402929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs") pod "network-metrics-daemon-b5zcw" (UID: "bf6b5042-e40a-4549-a761-2ef68ad8c71b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 19:56:39.486708 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.486696    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret podName:c642abb8-6e39-49db-955a-442707a3f0a6 nodeName:}" failed. No retries permitted until 2026-07-17 19:57:11.486673926 +0000 UTC m=+66.180650062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret") pod "global-pull-secret-syncer-mdlgw" (UID: "c642abb8-6e39-49db-955a-442707a3f0a6") : object "kube-system"/"original-pull-secret" not registered
Jul 17 19:56:39.587267 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:39.587226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:39.587408 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.587370    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 19:56:39.587408 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.587387    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 19:56:39.587408 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.587396    2572 projected.go:196] Error preparing data for projected volume kube-api-access-fb5qf for pod openshift-network-diagnostics/network-check-target-stgxg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:39.587511 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.587440    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf podName:5ddde34f-c002-4405-a98d-ce52d47a9bdb nodeName:}" failed. No retries permitted until 2026-07-17 19:57:11.587426847 +0000 UTC m=+66.281402972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fb5qf" (UniqueName: "kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf") pod "network-check-target-stgxg" (UID: "5ddde34f-c002-4405-a98d-ce52d47a9bdb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 19:56:39.847653 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:39.847560    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:39.847812 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:39.847560    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:39.847812 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.847687    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b5zcw" podUID="bf6b5042-e40a-4549-a761-2ef68ad8c71b"
Jul 17 19:56:39.847812 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:39.847569    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:39.847812 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.847786    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdlgw" podUID="c642abb8-6e39-49db-955a-442707a3f0a6"
Jul 17 19:56:39.848023 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:39.847833    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stgxg" podUID="5ddde34f-c002-4405-a98d-ce52d47a9bdb"
Jul 17 19:56:40.058861 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.058829    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-185.ec2.internal" event="NodeReady"
Jul 17 19:56:40.059039 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.059001    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 19:56:40.109383 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.109313    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9n9c2"]
Jul 17 19:56:40.112011 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.111990    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pdldf"]
Jul 17 19:56:40.112191 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.112169    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.113939 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.113920    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fd568b686-w7zvj"]
Jul 17 19:56:40.114063 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.114031    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.114440 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.114422    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 19:56:40.114550 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.114533    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 19:56:40.114611 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.114583    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wkppg\""
Jul 17 19:56:40.115528 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.115512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.118234 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.117848    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 19:56:40.118234 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.117988    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 19:56:40.118234 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.118032    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 19:56:40.118234 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.118136    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c78df\""
Jul 17 19:56:40.118234 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.118189    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 19:56:40.118522 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.118274    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wqfls\""
Jul 17 19:56:40.118709 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.118691    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 19:56:40.118804 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.118692    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 19:56:40.122622 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.122597    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pdldf"]
Jul 17 19:56:40.124866 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.124804    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 19:56:40.125854 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.125835    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9n9c2"]
Jul 17 19:56:40.129153 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.129135    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd568b686-w7zvj"]
Jul 17 19:56:40.291722 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0-cert\") pod \"ingress-canary-pdldf\" (UID: \"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0\") " pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.291722 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291736    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqqz5\" (UniqueName: \"kubernetes.io/projected/cf391883-f0be-4ab2-a2dd-309cd902f61b-kube-api-access-nqqz5\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.291967 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2830cc2d-e43d-4d0b-a942-326192a49580-ca-trust-extracted\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.291967 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf391883-f0be-4ab2-a2dd-309cd902f61b-config-volume\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.291967 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ft7q\" (UniqueName: \"kubernetes.io/projected/0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0-kube-api-access-2ft7q\") pod \"ingress-canary-pdldf\" (UID: \"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0\") " pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.291967 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291942    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.291967 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgpp\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-kube-api-access-hmgpp\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.292244 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.291979    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-trusted-ca\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.292244 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.292006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-registry-certificates\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.292244 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.292116    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf391883-f0be-4ab2-a2dd-309cd902f61b-tmp-dir\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.292244 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.292138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-image-registry-private-configuration\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.292244 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.292156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.292244 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.292183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-bound-sa-token\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.292244 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.292220    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-installation-pull-secrets\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393116 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.393116 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgpp\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-kube-api-access-hmgpp\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393116 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393120    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-trusted-ca\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-registry-certificates\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf391883-f0be-4ab2-a2dd-309cd902f61b-tmp-dir\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.393197    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393203    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-image-registry-private-configuration\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.393268    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls podName:cf391883-f0be-4ab2-a2dd-309cd902f61b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:40.893248155 +0000 UTC m=+35.587224282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls") pod "dns-default-9n9c2" (UID: "cf391883-f0be-4ab2-a2dd-309cd902f61b") : secret "dns-default-metrics-tls" not found
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393303    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393337    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-bound-sa-token\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393378 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393369    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-installation-pull-secrets\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393791 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393397    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0-cert\") pod \"ingress-canary-pdldf\" (UID: \"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0\") " pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.393791 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393416    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqqz5\" (UniqueName: \"kubernetes.io/projected/cf391883-f0be-4ab2-a2dd-309cd902f61b-kube-api-access-nqqz5\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.393791 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393431    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2830cc2d-e43d-4d0b-a942-326192a49580-ca-trust-extracted\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.393791 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393464    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf391883-f0be-4ab2-a2dd-309cd902f61b-config-volume\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.393791 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ft7q\" (UniqueName: \"kubernetes.io/projected/0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0-kube-api-access-2ft7q\") pod \"ingress-canary-pdldf\" (UID: \"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0\") " pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.394022 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393792    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf391883-f0be-4ab2-a2dd-309cd902f61b-tmp-dir\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.394022 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.393809    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-registry-certificates\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.394022 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.393895    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 19:56:40.394022 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.393910    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd568b686-w7zvj: secret "image-registry-tls" not found
Jul 17 19:56:40.394022 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.393955    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls podName:2830cc2d-e43d-4d0b-a942-326192a49580 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:40.893938522 +0000 UTC m=+35.587914660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls") pod "image-registry-7fd568b686-w7zvj" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580") : secret "image-registry-tls" not found
Jul 17 19:56:40.394300 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.394218    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2830cc2d-e43d-4d0b-a942-326192a49580-ca-trust-extracted\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.394519 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.394447    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-trusted-ca\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.394670 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.394642    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf391883-f0be-4ab2-a2dd-309cd902f61b-config-volume\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.398388 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.398365    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-installation-pull-secrets\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.398507 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.398376    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0-cert\") pod \"ingress-canary-pdldf\" (UID: \"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0\") " pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.398507 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.398413    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-image-registry-private-configuration\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.419683 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.418356    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ft7q\" (UniqueName: \"kubernetes.io/projected/0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0-kube-api-access-2ft7q\") pod \"ingress-canary-pdldf\" (UID: \"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0\") " pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.419683 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.418406    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-bound-sa-token\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.419683 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.418508    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgpp\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-kube-api-access-hmgpp\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.420079 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.420040    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqqz5\" (UniqueName: \"kubernetes.io/projected/cf391883-f0be-4ab2-a2dd-309cd902f61b-kube-api-access-nqqz5\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.433574 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.433548    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pdldf"
Jul 17 19:56:40.897460 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.897420    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:40.897999 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:40.897519    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:40.897999 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.897601    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 19:56:40.897999 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.897622    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd568b686-w7zvj: secret "image-registry-tls" not found
Jul 17 19:56:40.897999 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.897641    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 19:56:40.897999 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.897678    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls podName:2830cc2d-e43d-4d0b-a942-326192a49580 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:41.89766142 +0000 UTC m=+36.591637544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls") pod "image-registry-7fd568b686-w7zvj" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580") : secret "image-registry-tls" not found
Jul 17 19:56:40.897999 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:40.897694    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls podName:cf391883-f0be-4ab2-a2dd-309cd902f61b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:41.897686051 +0000 UTC m=+36.591662175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls") pod "dns-default-9n9c2" (UID: "cf391883-f0be-4ab2-a2dd-309cd902f61b") : secret "dns-default-metrics-tls" not found
Jul 17 19:56:41.504304 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.504155    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pdldf"]
Jul 17 19:56:41.507400 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:56:41.507365    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8996f1_c95e_4e7b_9f2e_e8bd0b08ddd0.slice/crio-a5ae86107c1f480728ac08a38d37001b7fa0fc184105f6d3530b98f8a71dbf32 WatchSource:0}: Error finding container a5ae86107c1f480728ac08a38d37001b7fa0fc184105f6d3530b98f8a71dbf32: Status 404 returned error can't find the container with id a5ae86107c1f480728ac08a38d37001b7fa0fc184105f6d3530b98f8a71dbf32
Jul 17 19:56:41.847700 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.847663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:56:41.847888 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.847666    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:56:41.847888 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.847666    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:56:41.851064 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.851029    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 19:56:41.851194 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.851069    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 19:56:41.851369 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.851354    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdp2d\""
Jul 17 19:56:41.851433 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.851386    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clqd4\""
Jul 17 19:56:41.851940 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.851922    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 19:56:41.852022 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.852005    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 19:56:41.905289 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.905262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:41.905942 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:41.905302    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:41.905942 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:41.905402    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 19:56:41.905942 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:41.905408    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 19:56:41.905942 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:41.905480    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls podName:cf391883-f0be-4ab2-a2dd-309cd902f61b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:43.905459715 +0000 UTC m=+38.599435852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls") pod "dns-default-9n9c2" (UID: "cf391883-f0be-4ab2-a2dd-309cd902f61b") : secret "dns-default-metrics-tls" not found
Jul 17 19:56:41.905942 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:41.905411    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd568b686-w7zvj: secret "image-registry-tls" not found
Jul 17 19:56:41.905942 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:41.905531    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls podName:2830cc2d-e43d-4d0b-a942-326192a49580 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:43.905519341 +0000 UTC m=+38.599495469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls") pod "image-registry-7fd568b686-w7zvj" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580") : secret "image-registry-tls" not found
Jul 17 19:56:42.048468 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:42.048431    2572 generic.go:356] "Generic (PLEG): container finished" podID="c048b030-3ada-4681-b3d7-3f177fa5122c" containerID="84f591bf2d06516f3f5ba76ccf3d52bc31960afb8194422bf2ed30c62356ed1b" exitCode=0
Jul 17 19:56:42.048623 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:42.048513    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerDied","Data":"84f591bf2d06516f3f5ba76ccf3d52bc31960afb8194422bf2ed30c62356ed1b"}
Jul 17 19:56:42.049779 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:42.049755    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pdldf" event={"ID":"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0","Type":"ContainerStarted","Data":"a5ae86107c1f480728ac08a38d37001b7fa0fc184105f6d3530b98f8a71dbf32"}
Jul 17 19:56:43.056152 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:43.056117    2572 generic.go:356] "Generic (PLEG): container finished" podID="c048b030-3ada-4681-b3d7-3f177fa5122c" containerID="7f61ebd9e1a66272e18944803d2cf1846f45e82e71917478ec12b316b16ba9f8" exitCode=0
Jul 17 19:56:43.056590 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:43.056191    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerDied","Data":"7f61ebd9e1a66272e18944803d2cf1846f45e82e71917478ec12b316b16ba9f8"}
Jul 17 19:56:43.919900 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:43.919865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:43.920056 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:43.919918    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:43.920056 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:43.920028    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 19:56:43.920056 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:43.920031    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 19:56:43.920164 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:43.920111    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls podName:cf391883-f0be-4ab2-a2dd-309cd902f61b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:47.920095083 +0000 UTC m=+42.614071212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls") pod "dns-default-9n9c2" (UID: "cf391883-f0be-4ab2-a2dd-309cd902f61b") : secret "dns-default-metrics-tls" not found
Jul 17 19:56:43.920164 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:43.920038    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd568b686-w7zvj: secret "image-registry-tls" not found
Jul 17 19:56:43.920230 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:43.920172    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls podName:2830cc2d-e43d-4d0b-a942-326192a49580 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:47.920160566 +0000 UTC m=+42.614136695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls") pod "image-registry-7fd568b686-w7zvj" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580") : secret "image-registry-tls" not found
Jul 17 19:56:44.064915 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:44.064885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jt22m" event={"ID":"c048b030-3ada-4681-b3d7-3f177fa5122c","Type":"ContainerStarted","Data":"edfc79d422e847bdf3bf04a216868a22ee68bd0d4b950cab2503eaf743c61be8"}
Jul 17 19:56:44.087490 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:44.087444    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jt22m" podStartSLOduration=8.260218066 podStartE2EDuration="38.087430749s" totalImagesPullingTime="29.827212683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:44.086192272 +0000 UTC m=+38.780168418" watchObservedRunningTime="2026-07-17 19:56:44.087430749 +0000 UTC m=+38.781406934"
Jul 17 19:56:45.069529 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:45.069274    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pdldf" event={"ID":"0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0","Type":"ContainerStarted","Data":"91b57b2ab56384fd515e4f699fb218f0611541478b3492f73c55b004b44cccf7"}
Jul 17 19:56:45.084306 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:45.084264    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pdldf" podStartSLOduration=2.5342338890000002 podStartE2EDuration="5.084248633s" totalImagesPullingTime="2.550014744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:56:45.0833698 +0000 UTC m=+39.777345946" watchObservedRunningTime="2026-07-17 19:56:45.084248633 +0000 UTC m=+39.778224778"
Jul 17 19:56:47.947374 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:47.947334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:47.947712 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:47.947384    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:47.947712 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:47.947475    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 19:56:47.947712 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:47.947485    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd568b686-w7zvj: secret "image-registry-tls" not found
Jul 17 19:56:47.947712 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:47.947489    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 19:56:47.947712 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:47.947532    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls podName:2830cc2d-e43d-4d0b-a942-326192a49580 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:55.947519038 +0000 UTC m=+50.641495163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls") pod "image-registry-7fd568b686-w7zvj" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580") : secret "image-registry-tls" not found
Jul 17 19:56:47.947712 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:47.947544    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls podName:cf391883-f0be-4ab2-a2dd-309cd902f61b nodeName:}" failed. No retries permitted until 2026-07-17 19:56:55.947538665 +0000 UTC m=+50.641514788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls") pod "dns-default-9n9c2" (UID: "cf391883-f0be-4ab2-a2dd-309cd902f61b") : secret "dns-default-metrics-tls" not found
Jul 17 19:56:50.768315 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.768282    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jvcr7"]
Jul 17 19:56:50.800857 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.800828    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:50.804176 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.804144    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 19:56:50.804301 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.804194    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 19:56:50.804420 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.804404    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmtpw\""
Jul 17 19:56:50.804915 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.804896    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 19:56:50.805021 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.804947    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 19:56:50.806363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.806341    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvcr7"]
Jul 17 19:56:50.970648 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.970613    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f98e6187-4523-4d1b-862c-18bc65445b93-crio-socket\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:50.970648 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.970650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f98e6187-4523-4d1b-862c-18bc65445b93-data-volume\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:50.970852 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.970669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:50.970852 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.970687    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:50.970852 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:50.970822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqs77\" (UniqueName: \"kubernetes.io/projected/f98e6187-4523-4d1b-862c-18bc65445b93-kube-api-access-vqs77\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.071596 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.071503    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f98e6187-4523-4d1b-862c-18bc65445b93-crio-socket\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.071596 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.071546    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f98e6187-4523-4d1b-862c-18bc65445b93-data-volume\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.071596 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.071563    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.071798 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.071650    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 19:56:51.071798 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.071704    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:51.571689753 +0000 UTC m=+46.265665877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : configmap "kube-rbac-proxy" not found
Jul 17 19:56:51.071798 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.071727    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f98e6187-4523-4d1b-862c-18bc65445b93-crio-socket\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.071798 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.071778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.071925 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.071869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqs77\" (UniqueName: \"kubernetes.io/projected/f98e6187-4523-4d1b-862c-18bc65445b93-kube-api-access-vqs77\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.071958 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.071936    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:51.071992 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.071971    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f98e6187-4523-4d1b-862c-18bc65445b93-data-volume\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.072024 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.072001    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:51.571984264 +0000 UTC m=+46.265960388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:51.087815 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.087790    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqs77\" (UniqueName: \"kubernetes.io/projected/f98e6187-4523-4d1b-862c-18bc65445b93-kube-api-access-vqs77\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.575961 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.575921    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.575961 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:51.575962    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:51.576238 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.576080    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 19:56:51.576238 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.576147    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:51.576238 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.576173    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:52.576150284 +0000 UTC m=+47.270126421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : configmap "kube-rbac-proxy" not found
Jul 17 19:56:51.576238 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:51.576197    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:52.576180766 +0000 UTC m=+47.270156894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:52.497179 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:52.497153    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5z8z9_031d2f1f-e63a-4d02-ac4e-bd7a4eca7773/dns-node-resolver/0.log"
Jul 17 19:56:52.584098 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:52.584040    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:52.584255 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:52.584111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:52.584255 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:52.584134    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 19:56:52.584255 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:52.584199    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:54.584181251 +0000 UTC m=+49.278157392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : configmap "kube-rbac-proxy" not found
Jul 17 19:56:52.584255 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:52.584224    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:52.584395 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:52.584265    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:54.584254507 +0000 UTC m=+49.278230645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:53.903905 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:53.903877    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pjbwc_1e158eee-6883-41ae-996a-5dad5e07629b/node-ca/0.log"
Jul 17 19:56:54.300447 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:54.300419    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pdldf_0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0/serve-healthcheck-canary/0.log"
Jul 17 19:56:54.598604 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:54.598534    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:54.598604 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:54.598572    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:54.598750 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:54.598663    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 19:56:54.598750 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:54.598707    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:54.598750 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:54.598737    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:58.598718117 +0000 UTC m=+53.292694254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : configmap "kube-rbac-proxy" not found
Jul 17 19:56:54.598844 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:54.598754    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:56:58.598747563 +0000 UTC m=+53.292723687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:56.008284 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:56.008246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:56:56.008714 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:56.008323    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:56:56.008714 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:56.008410    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 19:56:56.008714 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:56.008432    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd568b686-w7zvj: secret "image-registry-tls" not found
Jul 17 19:56:56.008714 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:56.008436    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 19:56:56.008714 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:56.008485    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls podName:2830cc2d-e43d-4d0b-a942-326192a49580 nodeName:}" failed. No retries permitted until 2026-07-17 19:57:12.008469461 +0000 UTC m=+66.702445588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls") pod "image-registry-7fd568b686-w7zvj" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580") : secret "image-registry-tls" not found
Jul 17 19:56:56.008714 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:56.008499    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls podName:cf391883-f0be-4ab2-a2dd-309cd902f61b nodeName:}" failed. No retries permitted until 2026-07-17 19:57:12.008493138 +0000 UTC m=+66.702469266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls") pod "dns-default-9n9c2" (UID: "cf391883-f0be-4ab2-a2dd-309cd902f61b") : secret "dns-default-metrics-tls" not found
Jul 17 19:56:58.625250 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:58.625195    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:58.625250 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:58.625245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:56:58.625770 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:58.625395    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:58.625770 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:56:58.625470    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls podName:f98e6187-4523-4d1b-862c-18bc65445b93 nodeName:}" failed. No retries permitted until 2026-07-17 19:57:06.625452159 +0000 UTC m=+61.319428283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvcr7" (UID: "f98e6187-4523-4d1b-862c-18bc65445b93") : secret "insights-runtime-extractor-tls" not found
Jul 17 19:56:58.625874 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:56:58.625803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f98e6187-4523-4d1b-862c-18bc65445b93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:57:06.046013 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:06.045979    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9tgrl"
Jul 17 19:57:06.689975 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:06.689943    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:57:06.692193 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:06.692167    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f98e6187-4523-4d1b-862c-18bc65445b93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvcr7\" (UID: \"f98e6187-4523-4d1b-862c-18bc65445b93\") " pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:57:06.712769 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:06.712751    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmtpw\""
Jul 17 19:57:06.721165 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:06.721147    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvcr7"
Jul 17 19:57:06.873140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:06.873112    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvcr7"]
Jul 17 19:57:06.876191 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:06.876162    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98e6187_4523_4d1b_862c_18bc65445b93.slice/crio-2fd2e70d48eac38af7d04528b681a3d98cfc908d20930c30dd165d880963f967 WatchSource:0}: Error finding container 2fd2e70d48eac38af7d04528b681a3d98cfc908d20930c30dd165d880963f967: Status 404 returned error can't find the container with id 2fd2e70d48eac38af7d04528b681a3d98cfc908d20930c30dd165d880963f967
Jul 17 19:57:07.119274 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:07.119237    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvcr7" event={"ID":"f98e6187-4523-4d1b-862c-18bc65445b93","Type":"ContainerStarted","Data":"2fd2e70d48eac38af7d04528b681a3d98cfc908d20930c30dd165d880963f967"}
Jul 17 19:57:11.530026 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.529989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:57:11.530026 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.530035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:57:11.532393 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.532374    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 19:57:11.532472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.532394    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 19:57:11.542565 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.542539    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf6b5042-e40a-4549-a761-2ef68ad8c71b-metrics-certs\") pod \"network-metrics-daemon-b5zcw\" (UID: \"bf6b5042-e40a-4549-a761-2ef68ad8c71b\") " pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:57:11.542661 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.542539    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c642abb8-6e39-49db-955a-442707a3f0a6-original-pull-secret\") pod \"global-pull-secret-syncer-mdlgw\" (UID: \"c642abb8-6e39-49db-955a-442707a3f0a6\") " pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:57:11.558791 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.558773    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdp2d\""
Jul 17 19:57:11.567652 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.567630    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdlgw"
Jul 17 19:57:11.567835 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.567633    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b5zcw"
Jul 17 19:57:11.630832 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.630476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:57:11.632897 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.632871    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 19:57:11.644442 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.644279    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 19:57:11.655232 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.655203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5qf\" (UniqueName: \"kubernetes.io/projected/5ddde34f-c002-4405-a98d-ce52d47a9bdb-kube-api-access-fb5qf\") pod \"network-check-target-stgxg\" (UID: \"5ddde34f-c002-4405-a98d-ce52d47a9bdb\") " pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:57:11.704207 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.704178    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mdlgw"]
Jul 17 19:57:11.707247 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:11.707219    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc642abb8_6e39_49db_955a_442707a3f0a6.slice/crio-54e2456636a0bd627ee9a0f10e6cdf9775832f624f6ee6a23e29516cdeb3a758 WatchSource:0}: Error finding container 54e2456636a0bd627ee9a0f10e6cdf9775832f624f6ee6a23e29516cdeb3a758: Status 404 returned error can't find the container with id 54e2456636a0bd627ee9a0f10e6cdf9775832f624f6ee6a23e29516cdeb3a758
Jul 17 19:57:11.718007 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.717984    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b5zcw"]
Jul 17 19:57:11.720541 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:11.720518    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6b5042_e40a_4549_a761_2ef68ad8c71b.slice/crio-00c4de80edcd6a718d9f84d80547a882b43a3c1275c089969c0d48ec207d4d0b WatchSource:0}: Error finding container 00c4de80edcd6a718d9f84d80547a882b43a3c1275c089969c0d48ec207d4d0b: Status 404 returned error can't find the container with id 00c4de80edcd6a718d9f84d80547a882b43a3c1275c089969c0d48ec207d4d0b
Jul 17 19:57:11.865506 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.865440    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clqd4\""
Jul 17 19:57:11.874401 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.874378    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:57:11.995783 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:11.995759    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stgxg"]
Jul 17 19:57:11.997971 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:11.997946    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddde34f_c002_4405_a98d_ce52d47a9bdb.slice/crio-62f0d7c0625b5f601887f0ff08384f0b8af249ac34dd456068798a0f0047d569 WatchSource:0}: Error finding container 62f0d7c0625b5f601887f0ff08384f0b8af249ac34dd456068798a0f0047d569: Status 404 returned error can't find the container with id 62f0d7c0625b5f601887f0ff08384f0b8af249ac34dd456068798a0f0047d569
Jul 17 19:57:12.033820 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.033787    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:57:12.033955 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.033862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:57:12.036399 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.036374    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf391883-f0be-4ab2-a2dd-309cd902f61b-metrics-tls\") pod \"dns-default-9n9c2\" (UID: \"cf391883-f0be-4ab2-a2dd-309cd902f61b\") " pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:57:12.036399 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.036389    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"image-registry-7fd568b686-w7zvj\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") " pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:57:12.133457 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.133376    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b5zcw" event={"ID":"bf6b5042-e40a-4549-a761-2ef68ad8c71b","Type":"ContainerStarted","Data":"00c4de80edcd6a718d9f84d80547a882b43a3c1275c089969c0d48ec207d4d0b"}
Jul 17 19:57:12.135220 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.135192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvcr7" event={"ID":"f98e6187-4523-4d1b-862c-18bc65445b93","Type":"ContainerStarted","Data":"73790290aa336efa1318a10619e0d001744f602fb5c485c5283a86be2a2728db"}
Jul 17 19:57:12.136587 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.136560    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mdlgw" event={"ID":"c642abb8-6e39-49db-955a-442707a3f0a6","Type":"ContainerStarted","Data":"54e2456636a0bd627ee9a0f10e6cdf9775832f624f6ee6a23e29516cdeb3a758"}
Jul 17 19:57:12.137783 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.137753    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stgxg" event={"ID":"5ddde34f-c002-4405-a98d-ce52d47a9bdb","Type":"ContainerStarted","Data":"62f0d7c0625b5f601887f0ff08384f0b8af249ac34dd456068798a0f0047d569"}
Jul 17 19:57:12.228232 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.228196    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wkppg\""
Jul 17 19:57:12.236664 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.236599    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:57:12.241876 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.241852    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wqfls\""
Jul 17 19:57:12.250655 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.250632    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:57:12.398334 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.398248    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9n9c2"]
Jul 17 19:57:12.402004 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:12.401976    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf391883_f0be_4ab2_a2dd_309cd902f61b.slice/crio-16c4bf6d472ef8946748464f025f074a4a26c580b3313a1b819aba7ea7cd0d2f WatchSource:0}: Error finding container 16c4bf6d472ef8946748464f025f074a4a26c580b3313a1b819aba7ea7cd0d2f: Status 404 returned error can't find the container with id 16c4bf6d472ef8946748464f025f074a4a26c580b3313a1b819aba7ea7cd0d2f
Jul 17 19:57:12.419042 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:12.418226    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd568b686-w7zvj"]
Jul 17 19:57:12.422627 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:12.422586    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2830cc2d_e43d_4d0b_a942_326192a49580.slice/crio-de8350d48664cd8ff836ed9e584c44f5368863835c73a0ff206232c847b3cbea WatchSource:0}: Error finding container de8350d48664cd8ff836ed9e584c44f5368863835c73a0ff206232c847b3cbea: Status 404 returned error can't find the container with id de8350d48664cd8ff836ed9e584c44f5368863835c73a0ff206232c847b3cbea
Jul 17 19:57:13.143633 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:13.143593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n9c2" event={"ID":"cf391883-f0be-4ab2-a2dd-309cd902f61b","Type":"ContainerStarted","Data":"16c4bf6d472ef8946748464f025f074a4a26c580b3313a1b819aba7ea7cd0d2f"}
Jul 17 19:57:13.148140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:13.147923    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj" event={"ID":"2830cc2d-e43d-4d0b-a942-326192a49580","Type":"ContainerStarted","Data":"b3232da979888c72c1671e29c4b27e8af798a85e04ec08e28bf7fcd2046a4d16"}
Jul 17 19:57:13.148140 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:13.147974    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj" event={"ID":"2830cc2d-e43d-4d0b-a942-326192a49580","Type":"ContainerStarted","Data":"de8350d48664cd8ff836ed9e584c44f5368863835c73a0ff206232c847b3cbea"}
Jul 17 19:57:13.148834 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:13.148705    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:57:14.152870 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:14.152827    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b5zcw" event={"ID":"bf6b5042-e40a-4549-a761-2ef68ad8c71b","Type":"ContainerStarted","Data":"7818b830c72f3c2ae30c4f274e8b9dc37741db3e8108be4dbd9b4f1b0e0731b0"}
Jul 17 19:57:15.096436 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.096346    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj" podStartSLOduration=66.096320823 podStartE2EDuration="1m6.096320823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:13.169303725 +0000 UTC m=+67.863279871" watchObservedRunningTime="2026-07-17 19:57:15.096320823 +0000 UTC m=+69.790296971"
Jul 17 19:57:15.097002 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.096979    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"]
Jul 17 19:57:15.168177 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.167965    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"]
Jul 17 19:57:15.168177 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.168113    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.172395 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.172368    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 19:57:15.173341 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.172669    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 19:57:15.173341 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.172940    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 19:57:15.173341 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.173193    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 19:57:15.211113 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.210943    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"]
Jul 17 19:57:15.211113 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.211014    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"]
Jul 17 19:57:15.211113 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.211036    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd568b686-w7zvj"]
Jul 17 19:57:15.211113 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.211112    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.213598 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.213542    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 17 19:57:15.213598 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.213577    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 17 19:57:15.213934 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.213577    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 17 19:57:15.213934 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.213708    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 17 19:57:15.258131 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258079    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxf62\" (UniqueName: \"kubernetes.io/projected/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-kube-api-access-vxf62\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.258131 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258131    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.258363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258161    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-ca\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.258363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258190    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c54d92-7060-4d96-88bd-0cc427fa3116-tmp\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.258363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.258363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258256    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltl2j\" (UniqueName: \"kubernetes.io/projected/b1c54d92-7060-4d96-88bd-0cc427fa3116-kube-api-access-ltl2j\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.258619 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258361    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-hub\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.258619 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258418    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.258619 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.258475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1c54d92-7060-4d96-88bd-0cc427fa3116-klusterlet-config\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.358872 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxf62\" (UniqueName: \"kubernetes.io/projected/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-kube-api-access-vxf62\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.358872 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358830    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.358872 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358855    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-ca\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.358872 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358876    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c54d92-7060-4d96-88bd-0cc427fa3116-tmp\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.359204 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358892    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.359204 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358912    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltl2j\" (UniqueName: \"kubernetes.io/projected/b1c54d92-7060-4d96-88bd-0cc427fa3116-kube-api-access-ltl2j\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.359204 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-hub\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.359204 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.358979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.359204 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.359013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1c54d92-7060-4d96-88bd-0cc427fa3116-klusterlet-config\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.359454 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.359310    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c54d92-7060-4d96-88bd-0cc427fa3116-tmp\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.359641 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.359614    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.361818 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.361780    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-ca\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.361924 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.361877    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.361976 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.361934    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-hub\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.361976 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.361947    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1c54d92-7060-4d96-88bd-0cc427fa3116-klusterlet-config\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.362107 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.362017    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.367456 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.367434    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxf62\" (UniqueName: \"kubernetes.io/projected/bceed97f-208e-4ca7-bf0b-2c18d3fd8777-kube-api-access-vxf62\") pod \"cluster-proxy-proxy-agent-67bbf8cf76-q8fgk\" (UID: \"bceed97f-208e-4ca7-bf0b-2c18d3fd8777\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:15.367957 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.367928    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltl2j\" (UniqueName: \"kubernetes.io/projected/b1c54d92-7060-4d96-88bd-0cc427fa3116-kube-api-access-ltl2j\") pod \"klusterlet-addon-workmgr-89c58ff57-4hj9x\" (UID: \"b1c54d92-7060-4d96-88bd-0cc427fa3116\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.479986 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.479956    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:15.529228 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:15.529186    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"
Jul 17 19:57:19.070534 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.070503    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk"]
Jul 17 19:57:19.075634 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:19.075595    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbceed97f_208e_4ca7_bf0b_2c18d3fd8777.slice/crio-fb54cc09656a30933e60b19773db8d552b6e1a280958bf1dbc72721b3897dd6f WatchSource:0}: Error finding container fb54cc09656a30933e60b19773db8d552b6e1a280958bf1dbc72721b3897dd6f: Status 404 returned error can't find the container with id fb54cc09656a30933e60b19773db8d552b6e1a280958bf1dbc72721b3897dd6f
Jul 17 19:57:19.087388 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.087321    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"]
Jul 17 19:57:19.092206 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:19.091084    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c54d92_7060_4d96_88bd_0cc427fa3116.slice/crio-fd5df342e699a013b06c761a8406329fe27b0fedbcf93dc87f36fe0f42904e9b WatchSource:0}: Error finding container fd5df342e699a013b06c761a8406329fe27b0fedbcf93dc87f36fe0f42904e9b: Status 404 returned error can't find the container with id fd5df342e699a013b06c761a8406329fe27b0fedbcf93dc87f36fe0f42904e9b
Jul 17 19:57:19.174250 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.174165    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b5zcw" event={"ID":"bf6b5042-e40a-4549-a761-2ef68ad8c71b","Type":"ContainerStarted","Data":"ec07d6e2bc459d85924b5267ae666cda334ba6e728458274ed44298bf595cbdc"}
Jul 17 19:57:19.175497 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.175469    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk" event={"ID":"bceed97f-208e-4ca7-bf0b-2c18d3fd8777","Type":"ContainerStarted","Data":"fb54cc09656a30933e60b19773db8d552b6e1a280958bf1dbc72721b3897dd6f"}
Jul 17 19:57:19.176660 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.176639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x" event={"ID":"b1c54d92-7060-4d96-88bd-0cc427fa3116","Type":"ContainerStarted","Data":"fd5df342e699a013b06c761a8406329fe27b0fedbcf93dc87f36fe0f42904e9b"}
Jul 17 19:57:19.178120 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.178099    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvcr7" event={"ID":"f98e6187-4523-4d1b-862c-18bc65445b93","Type":"ContainerStarted","Data":"025b13e8861cc3231d4f3959c8b515ad25d4a5ca33966188ed62757683ed8bf6"}
Jul 17 19:57:19.179483 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.179464    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mdlgw" event={"ID":"c642abb8-6e39-49db-955a-442707a3f0a6","Type":"ContainerStarted","Data":"44ba95725a070429d472433e899593678ecb1175671c516d92bc9cc2777a73f1"}
Jul 17 19:57:19.181585 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.181564    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stgxg" event={"ID":"5ddde34f-c002-4405-a98d-ce52d47a9bdb","Type":"ContainerStarted","Data":"377247552a0fea3a65c732b8ae69bc663fcb6f9f1a50df035da2b643181570bd"}
Jul 17 19:57:19.181904 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.181890    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 19:57:19.183757 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.183735    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n9c2" event={"ID":"cf391883-f0be-4ab2-a2dd-309cd902f61b","Type":"ContainerStarted","Data":"4533b8b85ea4ce9c53bba1fbf7c8d6e822a1de73c3d5f20aef74da6d1b2cf269"}
Jul 17 19:57:19.183896 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.183882    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:57:19.197077 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.197019    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b5zcw" podStartSLOduration=71.34024575 podStartE2EDuration="1m13.197006657s" totalImagesPullingTime="1.856760907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:19.195997264 +0000 UTC m=+73.889973447" watchObservedRunningTime="2026-07-17 19:57:19.197006657 +0000 UTC m=+73.890983175"
Jul 17 19:57:19.212626 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.212584    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9n9c2" podStartSLOduration=32.748315471 podStartE2EDuration="39.212568912s" totalImagesPullingTime="6.464253441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:19.21201802 +0000 UTC m=+73.905994188" watchObservedRunningTime="2026-07-17 19:57:19.212568912 +0000 UTC m=+73.906545059"
Jul 17 19:57:19.226975 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.226651    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mdlgw" podStartSLOduration=66.04722438 podStartE2EDuration="1m13.226637521s" totalImagesPullingTime="7.179413141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:19.226412933 +0000 UTC m=+73.920389063" watchObservedRunningTime="2026-07-17 19:57:19.226637521 +0000 UTC m=+73.920613666"
Jul 17 19:57:19.242629 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:19.242584    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-stgxg" podStartSLOduration=66.373756691 podStartE2EDuration="1m13.242569332s" totalImagesPullingTime="6.868812641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:19.241581227 +0000 UTC m=+73.935557367" watchObservedRunningTime="2026-07-17 19:57:19.242569332 +0000 UTC m=+73.936545477"
Jul 17 19:57:20.195074 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:20.194903    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n9c2" event={"ID":"cf391883-f0be-4ab2-a2dd-309cd902f61b","Type":"ContainerStarted","Data":"245c5585115c1eff520c2bc4d9935620911768dee2c3e43c678fa24ebbadc0c8"}
Jul 17 19:57:26.225009 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.224975    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x" event={"ID":"b1c54d92-7060-4d96-88bd-0cc427fa3116","Type":"ContainerStarted","Data":"ef8ba5a5088d1123710532fe2700b55a63e8fa31661265358a708bccaecb4ae3"}
Jul 17 19:57:26.225408 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.225387    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:26.228141 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.227842    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvcr7" event={"ID":"f98e6187-4523-4d1b-862c-18bc65445b93","Type":"ContainerStarted","Data":"b9fd7aaa031dd3a3459fcf6308e3e353e9f1318fe9db8b55f487b8bfeb8b8918"}
Jul 17 19:57:26.228483 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.228467    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x"
Jul 17 19:57:26.229970 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.229943    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk" event={"ID":"bceed97f-208e-4ca7-bf0b-2c18d3fd8777","Type":"ContainerStarted","Data":"ca3bee26cc54561eb558e37f272eb699890370533fc63cf2ee13955ed59569e1"}
Jul 17 19:57:26.244968 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.244346    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-89c58ff57-4hj9x" podStartSLOduration=4.235330482 podStartE2EDuration="11.24432231s" totalImagesPullingTime="7.008991828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:26.243124604 +0000 UTC m=+80.937100750" watchObservedRunningTime="2026-07-17 19:57:26.24432231 +0000 UTC m=+80.938298455"
Jul 17 19:57:26.280912 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.280860    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jvcr7" podStartSLOduration=17.274975055 podStartE2EDuration="36.280840911s" totalImagesPullingTime="19.005865856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:56:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:26.28072352 +0000 UTC m=+80.974699679" watchObservedRunningTime="2026-07-17 19:57:26.280840911 +0000 UTC m=+80.974817058"
Jul 17 19:57:26.933168 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.933125    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"]
Jul 17 19:57:26.936034 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.936016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:26.938232 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.938209    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 17 19:57:26.939363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.939339    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 17 19:57:26.939363 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.939355    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 19:57:26.939517 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.939370    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 19:57:26.939517 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.939343    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 19:57:26.939517 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.939346    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5tb9l\""
Jul 17 19:57:26.950094 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:26.947063    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"]
Jul 17 19:57:27.052688 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.052649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.052833 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.052699    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4fr\" (UniqueName: \"kubernetes.io/projected/da4702ca-10a0-405c-b98d-41a8642b57c4-kube-api-access-xl4fr\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.052833 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.052759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.052833 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.052799    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da4702ca-10a0-405c-b98d-41a8642b57c4-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.154134 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.154098    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.154134 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.154142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4fr\" (UniqueName: \"kubernetes.io/projected/da4702ca-10a0-405c-b98d-41a8642b57c4-kube-api-access-xl4fr\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.154331 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.154167    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.154331 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.154201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da4702ca-10a0-405c-b98d-41a8642b57c4-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.154399 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:57:27.154338    2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 17 19:57:27.154455 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:57:27.154411    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-tls podName:da4702ca-10a0-405c-b98d-41a8642b57c4 nodeName:}" failed. No retries permitted until 2026-07-17 19:57:27.654391612 +0000 UTC m=+82.348367914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-wmzqn" (UID: "da4702ca-10a0-405c-b98d-41a8642b57c4") : secret "prometheus-operator-tls" not found
Jul 17 19:57:27.154873 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.154854    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da4702ca-10a0-405c-b98d-41a8642b57c4-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.156559 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.156537    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.162209 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.162188    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4fr\" (UniqueName: \"kubernetes.io/projected/da4702ca-10a0-405c-b98d-41a8642b57c4-kube-api-access-xl4fr\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.234814 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.234774    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk" event={"ID":"bceed97f-208e-4ca7-bf0b-2c18d3fd8777","Type":"ContainerStarted","Data":"f9c0ac3d73af86321ad7f164d4671d97c31e91133a857f62d87654094c744317"}
Jul 17 19:57:27.235277 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.234825    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk" event={"ID":"bceed97f-208e-4ca7-bf0b-2c18d3fd8777","Type":"ContainerStarted","Data":"83490c0323bef9265705c7faa5a440625978418fc3df8e5dd84f74164979e327"}
Jul 17 19:57:27.252131 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.252074    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bbf8cf76-q8fgk" podStartSLOduration=5.234322807 podStartE2EDuration="12.25203453s" totalImagesPullingTime="7.017711723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:27.251071499 +0000 UTC m=+81.945047650" watchObservedRunningTime="2026-07-17 19:57:27.25203453 +0000 UTC m=+81.946010723"
Jul 17 19:57:27.659501 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.659352    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.661771 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.661746    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da4702ca-10a0-405c-b98d-41a8642b57c4-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-wmzqn\" (UID: \"da4702ca-10a0-405c-b98d-41a8642b57c4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.846149 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.846103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"
Jul 17 19:57:27.973084 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:27.973039    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn"]
Jul 17 19:57:27.975741 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:27.975718    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4702ca_10a0_405c_b98d_41a8642b57c4.slice/crio-9cda6ea2c28115c1e8cc368e85cc8439d6368a8b1a8cd9733318b96f96219989 WatchSource:0}: Error finding container 9cda6ea2c28115c1e8cc368e85cc8439d6368a8b1a8cd9733318b96f96219989: Status 404 returned error can't find the container with id 9cda6ea2c28115c1e8cc368e85cc8439d6368a8b1a8cd9733318b96f96219989
Jul 17 19:57:28.239636 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:28.239590    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn" event={"ID":"da4702ca-10a0-405c-b98d-41a8642b57c4","Type":"ContainerStarted","Data":"9cda6ea2c28115c1e8cc368e85cc8439d6368a8b1a8cd9733318b96f96219989"}
Jul 17 19:57:30.202512 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:30.202478    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9n9c2"
Jul 17 19:57:30.249021 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:30.248981    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn" event={"ID":"da4702ca-10a0-405c-b98d-41a8642b57c4","Type":"ContainerStarted","Data":"d3932401d4f89ec1035524422d88c8c98aa8b4a52467ab423769354967c44eb3"}
Jul 17 19:57:30.249226 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:30.249071    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn" event={"ID":"da4702ca-10a0-405c-b98d-41a8642b57c4","Type":"ContainerStarted","Data":"ed4a1bc9fe62f4ff13eb615bae0eb4953632bf8672ddf286921c78bf989aa528"}
Jul 17 19:57:30.267032 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:30.266983    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wmzqn" podStartSLOduration=3.027501493 podStartE2EDuration="4.26696437s" totalImagesPullingTime="1.239462877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:30.265522335 +0000 UTC m=+84.959498482" watchObservedRunningTime="2026-07-17 19:57:30.26696437 +0000 UTC m=+84.960940517"
Jul 17 19:57:32.296780 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.296747    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pt9xq"]
Jul 17 19:57:32.299846 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.299829    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.302294 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.302273    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 19:57:32.302413 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.302375    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 19:57:32.302623 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.302583    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 19:57:32.302952 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.302936    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bmrll\""
Jul 17 19:57:32.392842 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.392801    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-textfile\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.392842 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.392846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-sys\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.393085 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.392880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-tls\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.393085 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.392927    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-accelerators-collector-config\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.393085 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.392976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-root\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.393085 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.393004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-wtmp\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.393085 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.393020    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20597f50-0946-417f-a0d3-a76a765a9572-metrics-client-ca\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.393085 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.393036    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xzz\" (UniqueName: \"kubernetes.io/projected/20597f50-0946-417f-a0d3-a76a765a9572-kube-api-access-r4xzz\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.393268 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.393112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494247 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494213    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494247 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-textfile\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494272    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-sys\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494308    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-tls\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-accelerators-collector-config\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494363    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-sys\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-root\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494404    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-root\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-wtmp\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494472 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20597f50-0946-417f-a0d3-a76a765a9572-metrics-client-ca\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494899 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494492    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xzz\" (UniqueName: \"kubernetes.io/projected/20597f50-0946-417f-a0d3-a76a765a9572-kube-api-access-r4xzz\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494899 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:57:32.494498    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 19:57:32.494899 ip-10-0-143-185 kubenswrapper[2572]: E0717 19:57:32.494570    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-tls podName:20597f50-0946-417f-a0d3-a76a765a9572 nodeName:}" failed. No retries permitted until 2026-07-17 19:57:32.994549087 +0000 UTC m=+87.688525211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-tls") pod "node-exporter-pt9xq" (UID: "20597f50-0946-417f-a0d3-a76a765a9572") : secret "node-exporter-tls" not found
Jul 17 19:57:32.494899 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494595    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-wtmp\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.494899 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494680    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-textfile\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.495173 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.494987    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-accelerators-collector-config\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.495173 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.495088    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20597f50-0946-417f-a0d3-a76a765a9572-metrics-client-ca\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.496530 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.496514    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.503130 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.503109    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xzz\" (UniqueName: \"kubernetes.io/projected/20597f50-0946-417f-a0d3-a76a765a9572-kube-api-access-r4xzz\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:32.998028 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:32.997982    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-tls\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:33.000286 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:33.000249    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20597f50-0946-417f-a0d3-a76a765a9572-node-exporter-tls\") pod \"node-exporter-pt9xq\" (UID: \"20597f50-0946-417f-a0d3-a76a765a9572\") " pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:33.208727 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:33.208692    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pt9xq"
Jul 17 19:57:33.220842 ip-10-0-143-185 kubenswrapper[2572]: W0717 19:57:33.220817    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20597f50_0946_417f_a0d3_a76a765a9572.slice/crio-487cf07810016c2c6a3568893fd5511b1b402450680d2d370f25fbbce2dc489a WatchSource:0}: Error finding container 487cf07810016c2c6a3568893fd5511b1b402450680d2d370f25fbbce2dc489a: Status 404 returned error can't find the container with id 487cf07810016c2c6a3568893fd5511b1b402450680d2d370f25fbbce2dc489a
Jul 17 19:57:33.260969 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:33.260885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pt9xq" event={"ID":"20597f50-0946-417f-a0d3-a76a765a9572","Type":"ContainerStarted","Data":"487cf07810016c2c6a3568893fd5511b1b402450680d2d370f25fbbce2dc489a"}
Jul 17 19:57:35.269329 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:35.269297    2572 generic.go:356] "Generic (PLEG): container finished" podID="20597f50-0946-417f-a0d3-a76a765a9572" containerID="c795be5075dd86d1c4f5e9e3098e27bb624184eac16320470a58bbc3769ac035" exitCode=0
Jul 17 19:57:35.269719 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:35.269387    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pt9xq" event={"ID":"20597f50-0946-417f-a0d3-a76a765a9572","Type":"ContainerDied","Data":"c795be5075dd86d1c4f5e9e3098e27bb624184eac16320470a58bbc3769ac035"}
Jul 17 19:57:36.164516 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:36.164489    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:57:36.274724 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:36.274682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pt9xq" event={"ID":"20597f50-0946-417f-a0d3-a76a765a9572","Type":"ContainerStarted","Data":"fd5f44b0194db8096f5166aaf361ab448cf29a60d84290c1085a1a3202058a59"}
Jul 17 19:57:36.275117 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:36.274742    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pt9xq" event={"ID":"20597f50-0946-417f-a0d3-a76a765a9572","Type":"ContainerStarted","Data":"480cdbf48b4e1cb4a379ac401218487fa1cf501933f8d06c4a9585ed6a67c361"}
Jul 17 19:57:36.293500 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:36.293447    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pt9xq" podStartSLOduration=3.153965397 podStartE2EDuration="4.293426094s" totalImagesPullingTime="1.076054315s" totalInitContainerRuntime="63.406382ms" isStatefulPod=true podCreationTimestamp="2026-07-17 19:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:57:36.29317831 +0000 UTC m=+90.987154457" watchObservedRunningTime="2026-07-17 19:57:36.293426094 +0000 UTC m=+90.987402242"
Jul 17 19:57:41.179369 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.179299    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj" podUID="2830cc2d-e43d-4d0b-a942-326192a49580" containerName="registry" containerID="cri-o://b3232da979888c72c1671e29c4b27e8af798a85e04ec08e28bf7fcd2046a4d16" gracePeriod=30
Jul 17 19:57:41.294423 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.294395    2572 generic.go:356] "Generic (PLEG): container finished" podID="2830cc2d-e43d-4d0b-a942-326192a49580" containerID="b3232da979888c72c1671e29c4b27e8af798a85e04ec08e28bf7fcd2046a4d16" exitCode=0
Jul 17 19:57:41.294535 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.294457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj" event={"ID":"2830cc2d-e43d-4d0b-a942-326192a49580","Type":"ContainerDied","Data":"b3232da979888c72c1671e29c4b27e8af798a85e04ec08e28bf7fcd2046a4d16"}
Jul 17 19:57:41.436349 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.436284    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:57:41.563236 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563198    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmgpp\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-kube-api-access-hmgpp\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563236 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563232    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563453 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563260    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-trusted-ca\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563453 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563276    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-bound-sa-token\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563453 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563307    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-image-registry-private-configuration\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563453 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563348    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-registry-certificates\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563453 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563377    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2830cc2d-e43d-4d0b-a942-326192a49580-ca-trust-extracted\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563453 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563401    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-installation-pull-secrets\") pod \"2830cc2d-e43d-4d0b-a942-326192a49580\" (UID: \"2830cc2d-e43d-4d0b-a942-326192a49580\") "
Jul 17 19:57:41.563730 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.563652    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:57:41.564144 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.564112    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:57:41.566099 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.566068    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:57:41.566217 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.566105    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:57:41.566217 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.566135    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:57:41.566217 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.566190    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-kube-api-access-hmgpp" (OuterVolumeSpecName: "kube-api-access-hmgpp") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "kube-api-access-hmgpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:57:41.566345 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.566251    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:57:41.574367 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.574336    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2830cc2d-e43d-4d0b-a942-326192a49580-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2830cc2d-e43d-4d0b-a942-326192a49580" (UID: "2830cc2d-e43d-4d0b-a942-326192a49580"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:57:41.664952 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.664904    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmgpp\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-kube-api-access-hmgpp\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:41.664952 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.664949    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-registry-tls\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:41.665166 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.664964    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-trusted-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:41.665166 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.664976    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2830cc2d-e43d-4d0b-a942-326192a49580-bound-sa-token\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:41.665166 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.664992    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-image-registry-private-configuration\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:41.665166 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.665005    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2830cc2d-e43d-4d0b-a942-326192a49580-registry-certificates\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:41.665166 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.665018    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2830cc2d-e43d-4d0b-a942-326192a49580-ca-trust-extracted\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:41.665166 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:41.665030    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2830cc2d-e43d-4d0b-a942-326192a49580-installation-pull-secrets\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 19:57:42.299702 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:42.299674    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj"
Jul 17 19:57:42.300188 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:42.299667    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd568b686-w7zvj" event={"ID":"2830cc2d-e43d-4d0b-a942-326192a49580","Type":"ContainerDied","Data":"de8350d48664cd8ff836ed9e584c44f5368863835c73a0ff206232c847b3cbea"}
Jul 17 19:57:42.300935 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:42.300914    2572 scope.go:117] "RemoveContainer" containerID="b3232da979888c72c1671e29c4b27e8af798a85e04ec08e28bf7fcd2046a4d16"
Jul 17 19:57:42.317108 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:42.317069    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd568b686-w7zvj"]
Jul 17 19:57:42.320518 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:42.320482    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fd568b686-w7zvj"]
Jul 17 19:57:43.852238 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:43.852205    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2830cc2d-e43d-4d0b-a942-326192a49580" path="/var/lib/kubelet/pods/2830cc2d-e43d-4d0b-a942-326192a49580/volumes"
Jul 17 19:57:51.202116 ip-10-0-143-185 kubenswrapper[2572]: I0717 19:57:51.202078    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-stgxg"
Jul 17 20:01:05.725143 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:05.725119    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 20:01:38.652341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.652301    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-t2wcx"]
Jul 17 20:01:38.653848 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.652519    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2830cc2d-e43d-4d0b-a942-326192a49580" containerName="registry"
Jul 17 20:01:38.653848 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.652528    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2830cc2d-e43d-4d0b-a942-326192a49580" containerName="registry"
Jul 17 20:01:38.653848 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.652578    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2830cc2d-e43d-4d0b-a942-326192a49580" containerName="registry"
Jul 17 20:01:38.654290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.654276    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:38.656643 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.656620    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 20:01:38.656761 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.656659    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 20:01:38.657474 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.657458    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vthfz\""
Jul 17 20:01:38.662674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.662654    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-t2wcx"]
Jul 17 20:01:38.794704 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.794667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8664e44-88a5-4e0c-9af0-0281fc3856bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-t2wcx\" (UID: \"d8664e44-88a5-4e0c-9af0-0281fc3856bf\") " pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:38.794704 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.794705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgttl\" (UniqueName: \"kubernetes.io/projected/d8664e44-88a5-4e0c-9af0-0281fc3856bf-kube-api-access-lgttl\") pod \"cert-manager-55b4d954d5-t2wcx\" (UID: \"d8664e44-88a5-4e0c-9af0-0281fc3856bf\") " pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:38.895265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.895229    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8664e44-88a5-4e0c-9af0-0281fc3856bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-t2wcx\" (UID: \"d8664e44-88a5-4e0c-9af0-0281fc3856bf\") " pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:38.895265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.895266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgttl\" (UniqueName: \"kubernetes.io/projected/d8664e44-88a5-4e0c-9af0-0281fc3856bf-kube-api-access-lgttl\") pod \"cert-manager-55b4d954d5-t2wcx\" (UID: \"d8664e44-88a5-4e0c-9af0-0281fc3856bf\") " pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:38.903516 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.903450    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgttl\" (UniqueName: \"kubernetes.io/projected/d8664e44-88a5-4e0c-9af0-0281fc3856bf-kube-api-access-lgttl\") pod \"cert-manager-55b4d954d5-t2wcx\" (UID: \"d8664e44-88a5-4e0c-9af0-0281fc3856bf\") " pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:38.903516 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.903451    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8664e44-88a5-4e0c-9af0-0281fc3856bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-t2wcx\" (UID: \"d8664e44-88a5-4e0c-9af0-0281fc3856bf\") " pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:38.963571 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:38.963540    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-t2wcx"
Jul 17 20:01:39.084088 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:39.084037    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-t2wcx"]
Jul 17 20:01:39.086824 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:01:39.086794    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8664e44_88a5_4e0c_9af0_0281fc3856bf.slice/crio-be7584bc4a2d35b952be84ed2547f1dafb4b9339a5ce7ce897bb27cc7e4baf7a WatchSource:0}: Error finding container be7584bc4a2d35b952be84ed2547f1dafb4b9339a5ce7ce897bb27cc7e4baf7a: Status 404 returned error can't find the container with id be7584bc4a2d35b952be84ed2547f1dafb4b9339a5ce7ce897bb27cc7e4baf7a
Jul 17 20:01:39.088676 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:39.088663    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 20:01:40.059439 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:40.059398    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-t2wcx" event={"ID":"d8664e44-88a5-4e0c-9af0-0281fc3856bf","Type":"ContainerStarted","Data":"be7584bc4a2d35b952be84ed2547f1dafb4b9339a5ce7ce897bb27cc7e4baf7a"}
Jul 17 20:01:43.070750 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:43.070718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-t2wcx" event={"ID":"d8664e44-88a5-4e0c-9af0-0281fc3856bf","Type":"ContainerStarted","Data":"9db44f76628c402ab9210968da7b4f712115115d67b4679e47ccdfcce75db1a6"}
Jul 17 20:01:43.086231 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:01:43.086179    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-t2wcx" podStartSLOduration=2.151074789 podStartE2EDuration="5.086163962s" totalImagesPullingTime="2.935089173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:01:43.085569237 +0000 UTC m=+337.779545383" watchObservedRunningTime="2026-07-17 20:01:43.086163962 +0000 UTC m=+337.780140108"
Jul 17 20:02:04.526558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.526469    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"]
Jul 17 20:02:04.530165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.530143    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.532567 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.532547    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 20:02:04.532865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.532843    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 20:02:04.532967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.532843    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rss4j\""
Jul 17 20:02:04.533331 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.533311    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 20:02:04.533657 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.533641    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 20:02:04.545802 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.545781    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"]
Jul 17 20:02:04.683010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.682965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.683010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.683020    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.683243 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.683038    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4d9\" (UniqueName: \"kubernetes.io/projected/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-kube-api-access-kv4d9\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.683243 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.683156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-tmp\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.683243 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.683204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-manifests\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.783888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.783798    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-tmp\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.783888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.783844    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-manifests\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.783888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.783874    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.784137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.783915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.784137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.783939    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4d9\" (UniqueName: \"kubernetes.io/projected/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-kube-api-access-kv4d9\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.784336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.784302    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-manifests\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.786279 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.786253    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-tmp\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.786380 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.786345    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.786492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.786463    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.791698 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.791674    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4d9\" (UniqueName: \"kubernetes.io/projected/74343d95-b829-4a9d-a2c5-35b7dc48c6b4-kube-api-access-kv4d9\") pod \"opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm\" (UID: \"74343d95-b829-4a9d-a2c5-35b7dc48c6b4\") " pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.840595 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.840558    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:04.976989 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:04.976959    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"]
Jul 17 20:02:04.980243 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:02:04.980207    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74343d95_b829_4a9d_a2c5_35b7dc48c6b4.slice/crio-554a8a0e8f36624d1bd9a17756a5a64574412c79ca93438c129e81e09d6689a1 WatchSource:0}: Error finding container 554a8a0e8f36624d1bd9a17756a5a64574412c79ca93438c129e81e09d6689a1: Status 404 returned error can't find the container with id 554a8a0e8f36624d1bd9a17756a5a64574412c79ca93438c129e81e09d6689a1
Jul 17 20:02:05.142812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:05.142717    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm" event={"ID":"74343d95-b829-4a9d-a2c5-35b7dc48c6b4","Type":"ContainerStarted","Data":"554a8a0e8f36624d1bd9a17756a5a64574412c79ca93438c129e81e09d6689a1"}
Jul 17 20:02:08.156257 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:08.156217    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm" event={"ID":"74343d95-b829-4a9d-a2c5-35b7dc48c6b4","Type":"ContainerStarted","Data":"5a12b1b4af7f41ccd7419964cfdcc8ab39729bdd7342e75b88d5a3738443af71"}
Jul 17 20:02:09.160907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:09.160868    2572 generic.go:356] "Generic (PLEG): container finished" podID="74343d95-b829-4a9d-a2c5-35b7dc48c6b4" containerID="5a12b1b4af7f41ccd7419964cfdcc8ab39729bdd7342e75b88d5a3738443af71" exitCode=0
Jul 17 20:02:09.161319 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:09.160954    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm" event={"ID":"74343d95-b829-4a9d-a2c5-35b7dc48c6b4","Type":"ContainerDied","Data":"5a12b1b4af7f41ccd7419964cfdcc8ab39729bdd7342e75b88d5a3738443af71"}
Jul 17 20:02:10.167360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:10.167326    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm" event={"ID":"74343d95-b829-4a9d-a2c5-35b7dc48c6b4","Type":"ContainerStarted","Data":"7c5edcf8febb910ef41cbf693a7ce992c337c2dcb1e16cb1d4ca9dedaff1be43"}
Jul 17 20:02:10.167767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:10.167472    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:10.187972 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:10.187917    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm" podStartSLOduration=1.879060547 podStartE2EDuration="6.187899739s" totalImagesPullingTime="3.283356422s" totalInitContainerRuntime="1.02548277s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:02:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:02:10.186606602 +0000 UTC m=+364.880582748" watchObservedRunningTime="2026-07-17 20:02:10.187899739 +0000 UTC m=+364.881875885"
Jul 17 20:02:21.173699 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:21.173668    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm"
Jul 17 20:02:24.130581 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.130543    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mfjbf"]
Jul 17 20:02:24.138510 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.138487    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.142166 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.142128    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fhc7h\""
Jul 17 20:02:24.142293 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.142223    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 17 20:02:24.148471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.148445    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mfjbf"]
Jul 17 20:02:24.214172 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.214144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-cert\") pod \"odh-model-controller-7998c67dc4-mfjbf\" (UID: \"ad0bb1e5-caf8-40cb-a9b8-7013d746e028\") " pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.214338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.214176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4c77\" (UniqueName: \"kubernetes.io/projected/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-kube-api-access-b4c77\") pod \"odh-model-controller-7998c67dc4-mfjbf\" (UID: \"ad0bb1e5-caf8-40cb-a9b8-7013d746e028\") " pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.314835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.314804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-cert\") pod \"odh-model-controller-7998c67dc4-mfjbf\" (UID: \"ad0bb1e5-caf8-40cb-a9b8-7013d746e028\") " pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.314835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.314838    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4c77\" (UniqueName: \"kubernetes.io/projected/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-kube-api-access-b4c77\") pod \"odh-model-controller-7998c67dc4-mfjbf\" (UID: \"ad0bb1e5-caf8-40cb-a9b8-7013d746e028\") " pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.315087 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:02:24.314968    2572 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 17 20:02:24.315087 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:02:24.315029    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-cert podName:ad0bb1e5-caf8-40cb-a9b8-7013d746e028 nodeName:}" failed. No retries permitted until 2026-07-17 20:02:24.815012845 +0000 UTC m=+379.508988970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-cert") pod "odh-model-controller-7998c67dc4-mfjbf" (UID: "ad0bb1e5-caf8-40cb-a9b8-7013d746e028") : secret "odh-model-controller-webhook-cert" not found
Jul 17 20:02:24.323420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.323395    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4c77\" (UniqueName: \"kubernetes.io/projected/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-kube-api-access-b4c77\") pod \"odh-model-controller-7998c67dc4-mfjbf\" (UID: \"ad0bb1e5-caf8-40cb-a9b8-7013d746e028\") " pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.508980 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.508948    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nhgzn"]
Jul 17 20:02:24.512099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.512081    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.514205 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.514182    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 17 20:02:24.525444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.525422    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nhgzn"]
Jul 17 20:02:24.617250 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.617219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjfm\" (UniqueName: \"kubernetes.io/projected/8e526790-634c-4370-8785-fa57241af7e9-kube-api-access-wtjfm\") pod \"model-serving-api-54d89bd5fc-nhgzn\" (UID: \"8e526790-634c-4370-8785-fa57241af7e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.617404 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.617257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e526790-634c-4370-8785-fa57241af7e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-nhgzn\" (UID: \"8e526790-634c-4370-8785-fa57241af7e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.718233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.718194    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjfm\" (UniqueName: \"kubernetes.io/projected/8e526790-634c-4370-8785-fa57241af7e9-kube-api-access-wtjfm\") pod \"model-serving-api-54d89bd5fc-nhgzn\" (UID: \"8e526790-634c-4370-8785-fa57241af7e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.718233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.718237    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e526790-634c-4370-8785-fa57241af7e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-nhgzn\" (UID: \"8e526790-634c-4370-8785-fa57241af7e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.720661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.720640    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e526790-634c-4370-8785-fa57241af7e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-nhgzn\" (UID: \"8e526790-634c-4370-8785-fa57241af7e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.725812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.725787    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjfm\" (UniqueName: \"kubernetes.io/projected/8e526790-634c-4370-8785-fa57241af7e9-kube-api-access-wtjfm\") pod \"model-serving-api-54d89bd5fc-nhgzn\" (UID: \"8e526790-634c-4370-8785-fa57241af7e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.819653 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.819558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-cert\") pod \"odh-model-controller-7998c67dc4-mfjbf\" (UID: \"ad0bb1e5-caf8-40cb-a9b8-7013d746e028\") " pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.821716 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.821681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:24.821994 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.821975    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad0bb1e5-caf8-40cb-a9b8-7013d746e028-cert\") pod \"odh-model-controller-7998c67dc4-mfjbf\" (UID: \"ad0bb1e5-caf8-40cb-a9b8-7013d746e028\") " pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:24.959685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:24.959652    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nhgzn"]
Jul 17 20:02:24.962474 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:02:24.962445    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e526790_634c_4370_8785_fa57241af7e9.slice/crio-89472d49bb7b0fff90cf5ab5d8e203c1ded5850c702d50032924ff08f785c7df WatchSource:0}: Error finding container 89472d49bb7b0fff90cf5ab5d8e203c1ded5850c702d50032924ff08f785c7df: Status 404 returned error can't find the container with id 89472d49bb7b0fff90cf5ab5d8e203c1ded5850c702d50032924ff08f785c7df
Jul 17 20:02:25.048332 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:25.048295    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:25.179227 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:25.179206    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mfjbf"]
Jul 17 20:02:25.182253 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:02:25.182212    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0bb1e5_caf8_40cb_a9b8_7013d746e028.slice/crio-40e675605dc168b54916b4fdb9591981f72e95c1426441858c068cafb7259b8c WatchSource:0}: Error finding container 40e675605dc168b54916b4fdb9591981f72e95c1426441858c068cafb7259b8c: Status 404 returned error can't find the container with id 40e675605dc168b54916b4fdb9591981f72e95c1426441858c068cafb7259b8c
Jul 17 20:02:25.223169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:25.223135    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn" event={"ID":"8e526790-634c-4370-8785-fa57241af7e9","Type":"ContainerStarted","Data":"89472d49bb7b0fff90cf5ab5d8e203c1ded5850c702d50032924ff08f785c7df"}
Jul 17 20:02:25.224304 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:25.224281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf" event={"ID":"ad0bb1e5-caf8-40cb-a9b8-7013d746e028","Type":"ContainerStarted","Data":"40e675605dc168b54916b4fdb9591981f72e95c1426441858c068cafb7259b8c"}
Jul 17 20:02:27.235899 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:27.235849    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn" event={"ID":"8e526790-634c-4370-8785-fa57241af7e9","Type":"ContainerStarted","Data":"5087ffcbba2aa3ca9931b4e42d6cbe2e6adbf8ea2964895540c14a2a21af0642"}
Jul 17 20:02:27.236365 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:27.236145    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:27.251582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:27.251529    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn" podStartSLOduration=1.397367198 podStartE2EDuration="3.2515143s" totalImagesPullingTime="1.854147102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:02:27.251093781 +0000 UTC m=+381.945069929" watchObservedRunningTime="2026-07-17 20:02:27.2515143 +0000 UTC m=+381.945490458"
Jul 17 20:02:29.245516 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:29.245475    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf" event={"ID":"ad0bb1e5-caf8-40cb-a9b8-7013d746e028","Type":"ContainerStarted","Data":"c894580d05577d0f6bfcf92b0098c5a67e0c4a00fd225470cead645dfda70122"}
Jul 17 20:02:29.246006 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:29.245617    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:29.265299 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:29.265229    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf" podStartSLOduration=2.212409953 podStartE2EDuration="5.265207519s" totalImagesPullingTime="3.052797566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:02:29.26300172 +0000 UTC m=+383.956977866" watchObservedRunningTime="2026-07-17 20:02:29.265207519 +0000 UTC m=+383.959183666"
Jul 17 20:02:33.684181 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.684145    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"]
Jul 17 20:02:33.687791 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.687766    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:33.690327 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.690306    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qq468\""
Jul 17 20:02:33.690691 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.690673    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 17 20:02:33.696414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.696391    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"]
Jul 17 20:02:33.699185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.699166    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"]
Jul 17 20:02:33.703720 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.703693    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:33.706385 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.706357    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5kn6n\""
Jul 17 20:02:33.706385 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.706380    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 17 20:02:33.714087 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.714034    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"]
Jul 17 20:02:33.788597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.788556    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqw8r\" (UniqueName: \"kubernetes.io/projected/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-kube-api-access-dqw8r\") pod \"kserve-controller-manager-d6f4b65b8-s5fvl\" (UID: \"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:33.788597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.788598    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9666413-1a40-4ad3-b2bc-113de3eb747f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxc4f\" (UID: \"e9666413-1a40-4ad3-b2bc-113de3eb747f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:33.788829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.788655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwd4\" (UniqueName: \"kubernetes.io/projected/e9666413-1a40-4ad3-b2bc-113de3eb747f-kube-api-access-hvwd4\") pod \"llmisvc-controller-manager-777c6bd7d5-bxc4f\" (UID: \"e9666413-1a40-4ad3-b2bc-113de3eb747f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:33.788829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.788726    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-cert\") pod \"kserve-controller-manager-d6f4b65b8-s5fvl\" (UID: \"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:33.889872 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.889836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqw8r\" (UniqueName: \"kubernetes.io/projected/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-kube-api-access-dqw8r\") pod \"kserve-controller-manager-d6f4b65b8-s5fvl\" (UID: \"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:33.889872 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.889876    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9666413-1a40-4ad3-b2bc-113de3eb747f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxc4f\" (UID: \"e9666413-1a40-4ad3-b2bc-113de3eb747f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:33.890151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.889905    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwd4\" (UniqueName: \"kubernetes.io/projected/e9666413-1a40-4ad3-b2bc-113de3eb747f-kube-api-access-hvwd4\") pod \"llmisvc-controller-manager-777c6bd7d5-bxc4f\" (UID: \"e9666413-1a40-4ad3-b2bc-113de3eb747f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:33.890151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.889950    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-cert\") pod \"kserve-controller-manager-d6f4b65b8-s5fvl\" (UID: \"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:33.890151 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:02:33.889998    2572 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 17 20:02:33.890151 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:02:33.890038    2572 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 17 20:02:33.890151 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:02:33.890109    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9666413-1a40-4ad3-b2bc-113de3eb747f-cert podName:e9666413-1a40-4ad3-b2bc-113de3eb747f nodeName:}" failed. No retries permitted until 2026-07-17 20:02:34.390086127 +0000 UTC m=+389.084062263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9666413-1a40-4ad3-b2bc-113de3eb747f-cert") pod "llmisvc-controller-manager-777c6bd7d5-bxc4f" (UID: "e9666413-1a40-4ad3-b2bc-113de3eb747f") : secret "llmisvc-webhook-server-cert" not found
Jul 17 20:02:33.890151 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:02:33.890125    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-cert podName:5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9 nodeName:}" failed. No retries permitted until 2026-07-17 20:02:34.390118813 +0000 UTC m=+389.084094937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-cert") pod "kserve-controller-manager-d6f4b65b8-s5fvl" (UID: "5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9") : secret "kserve-webhook-server-cert" not found
Jul 17 20:02:33.906708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.906675    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqw8r\" (UniqueName: \"kubernetes.io/projected/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-kube-api-access-dqw8r\") pod \"kserve-controller-manager-d6f4b65b8-s5fvl\" (UID: \"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:33.909378 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:33.909357    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwd4\" (UniqueName: \"kubernetes.io/projected/e9666413-1a40-4ad3-b2bc-113de3eb747f-kube-api-access-hvwd4\") pod \"llmisvc-controller-manager-777c6bd7d5-bxc4f\" (UID: \"e9666413-1a40-4ad3-b2bc-113de3eb747f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:34.394309 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.394270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9666413-1a40-4ad3-b2bc-113de3eb747f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxc4f\" (UID: \"e9666413-1a40-4ad3-b2bc-113de3eb747f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:34.394516 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.394336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-cert\") pod \"kserve-controller-manager-d6f4b65b8-s5fvl\" (UID: \"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:34.396789 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.396763    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9-cert\") pod \"kserve-controller-manager-d6f4b65b8-s5fvl\" (UID: \"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:34.396879 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.396776    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9666413-1a40-4ad3-b2bc-113de3eb747f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxc4f\" (UID: \"e9666413-1a40-4ad3-b2bc-113de3eb747f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:34.472334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.472292    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"]
Jul 17 20:02:34.475676 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.475656    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.478215 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.478194    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 20:02:34.478886 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.478857    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 20:02:34.478993 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.478888    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 20:02:34.478993 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.478889    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5wcdl\""
Jul 17 20:02:34.479198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.479181    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 20:02:34.486057 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.486024    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"]
Jul 17 20:02:34.595966 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.595923    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-tls-certs\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.595966 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.595963    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-tmp\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.596175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.595996    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6nhg\" (UniqueName: \"kubernetes.io/projected/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-kube-api-access-n6nhg\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.597915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.597899    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:34.617767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.617741    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:34.696681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.696641    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-tls-certs\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.697190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.697170    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-tmp\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.697374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.697356    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6nhg\" (UniqueName: \"kubernetes.io/projected/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-kube-api-access-n6nhg\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.699488 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.699463    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-tmp\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.699835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.699814    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-tls-certs\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.706773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.706743    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6nhg\" (UniqueName: \"kubernetes.io/projected/be6d4ef4-252f-4dd5-9570-3eacd013e3b3-kube-api-access-n6nhg\") pod \"kube-auth-proxy-fdbd865b7-2zgvv\" (UID: \"be6d4ef4-252f-4dd5-9570-3eacd013e3b3\") " pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.741691 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.741663    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"]
Jul 17 20:02:34.744293 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:02:34.744261    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5c686c_fea2_4dd0_98e5_2f2549bbdfb9.slice/crio-3b18cfa468f1996681d6219b3b3863a7c02b80597fe4b275fd2e7c29c6f7367c WatchSource:0}: Error finding container 3b18cfa468f1996681d6219b3b3863a7c02b80597fe4b275fd2e7c29c6f7367c: Status 404 returned error can't find the container with id 3b18cfa468f1996681d6219b3b3863a7c02b80597fe4b275fd2e7c29c6f7367c
Jul 17 20:02:34.764174 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.764147    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"]
Jul 17 20:02:34.767025 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:02:34.766994    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode9666413_1a40_4ad3_b2bc_113de3eb747f.slice/crio-d0db53665e3d2f89292dee6b2d85b80c5de10fbb830be153ab407fa5023f9469 WatchSource:0}: Error finding container d0db53665e3d2f89292dee6b2d85b80c5de10fbb830be153ab407fa5023f9469: Status 404 returned error can't find the container with id d0db53665e3d2f89292dee6b2d85b80c5de10fbb830be153ab407fa5023f9469
Jul 17 20:02:34.786083 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.786055    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"
Jul 17 20:02:34.927584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:34.927561    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv"]
Jul 17 20:02:34.929762 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:02:34.929739    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6d4ef4_252f_4dd5_9570_3eacd013e3b3.slice/crio-534fff521e9c60513bfe22b8765a8249f27efadf0aac2029034eab14cd16d02d WatchSource:0}: Error finding container 534fff521e9c60513bfe22b8765a8249f27efadf0aac2029034eab14cd16d02d: Status 404 returned error can't find the container with id 534fff521e9c60513bfe22b8765a8249f27efadf0aac2029034eab14cd16d02d
Jul 17 20:02:35.267256 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:35.267224    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv" event={"ID":"be6d4ef4-252f-4dd5-9570-3eacd013e3b3","Type":"ContainerStarted","Data":"534fff521e9c60513bfe22b8765a8249f27efadf0aac2029034eab14cd16d02d"}
Jul 17 20:02:35.268382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:35.268358    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f" event={"ID":"e9666413-1a40-4ad3-b2bc-113de3eb747f","Type":"ContainerStarted","Data":"d0db53665e3d2f89292dee6b2d85b80c5de10fbb830be153ab407fa5023f9469"}
Jul 17 20:02:35.269505 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:35.269479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl" event={"ID":"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9","Type":"ContainerStarted","Data":"3b18cfa468f1996681d6219b3b3863a7c02b80597fe4b275fd2e7c29c6f7367c"}
Jul 17 20:02:36.136176 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.136140    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"]
Jul 17 20:02:36.139730 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.139705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.142883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.142855    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 20:02:36.145446 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.145418    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4prv7\""
Jul 17 20:02:36.145676 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.145660    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 20:02:36.145832 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.145813    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 20:02:36.145914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.145835    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 20:02:36.160686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.160661    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"]
Jul 17 20:02:36.211216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.211181    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.211390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.211304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7pfn\" (UniqueName: \"kubernetes.io/projected/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-kube-api-access-l7pfn\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.211390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.211354    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.211512 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.211407    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.211512 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.211451    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.211512 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.211487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.211651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.211529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.313536 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.313490    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7pfn\" (UniqueName: \"kubernetes.io/projected/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-kube-api-access-l7pfn\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.313765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.313555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.313765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.313595    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.313765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.313671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.313765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.313716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.313765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.313761    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.314122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.313818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.316819 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.316603    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.318178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.318143    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.323068 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.323025    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7pfn\" (UniqueName: \"kubernetes.io/projected/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-kube-api-access-l7pfn\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.323472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.323435    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.324153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.324111    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.325288 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.325266    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.326990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.326959    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfa7f00-ae9c-4e06-a14d-937beeaa6381-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mfnl8\" (UID: \"dcfa7f00-ae9c-4e06-a14d-937beeaa6381\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.457790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.457707    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:36.763863 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:36.763715    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"]
Jul 17 20:02:36.769201 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:02:36.769161    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfa7f00_ae9c_4e06_a14d_937beeaa6381.slice/crio-c713c8fea86eb185b0fc8a946871bad67c5fce60117c0f218081e46dea32424f WatchSource:0}: Error finding container c713c8fea86eb185b0fc8a946871bad67c5fce60117c0f218081e46dea32424f: Status 404 returned error can't find the container with id c713c8fea86eb185b0fc8a946871bad67c5fce60117c0f218081e46dea32424f
Jul 17 20:02:37.280838 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:37.280792    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8" event={"ID":"dcfa7f00-ae9c-4e06-a14d-937beeaa6381","Type":"ContainerStarted","Data":"c713c8fea86eb185b0fc8a946871bad67c5fce60117c0f218081e46dea32424f"}
Jul 17 20:02:38.251364 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:38.251334    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-nhgzn"
Jul 17 20:02:39.291379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.291341    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f" event={"ID":"e9666413-1a40-4ad3-b2bc-113de3eb747f","Type":"ContainerStarted","Data":"20e7596491bcca0eeb8bbad8ae75aa2605acff34f9f101d8f89c8852c1cb505a"}
Jul 17 20:02:39.291817 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.291407    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:02:39.292913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.292885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv" event={"ID":"be6d4ef4-252f-4dd5-9570-3eacd013e3b3","Type":"ContainerStarted","Data":"c18c4257acb802b73f5fcec621b9252dbeb16aba7a4a439af9fea841b6c54fe4"}
Jul 17 20:02:39.294728 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.294700    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl" event={"ID":"5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9","Type":"ContainerStarted","Data":"8b3481a651b45853d1c8fc09b8d36c280bf27d1616550c650e46dd396a50b6b7"}
Jul 17 20:02:39.294830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.294769    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:02:39.308313 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.308265    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f" podStartSLOduration=2.170480215 podStartE2EDuration="6.308249325s" totalImagesPullingTime="4.13776911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:02:39.307434947 +0000 UTC m=+394.001411094" watchObservedRunningTime="2026-07-17 20:02:39.308249325 +0000 UTC m=+394.002225471"
Jul 17 20:02:39.326505 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.326449    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-fdbd865b7-2zgvv" podStartSLOduration=1.354937329 podStartE2EDuration="5.326433893s" totalImagesPullingTime="3.971496564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:02:39.325797928 +0000 UTC m=+394.019774087" watchObservedRunningTime="2026-07-17 20:02:39.326433893 +0000 UTC m=+394.020410038"
Jul 17 20:02:39.344865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:39.344818    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl" podStartSLOduration=2.191783772 podStartE2EDuration="6.344796311s" totalImagesPullingTime="4.153012539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:02:39.343526233 +0000 UTC m=+394.037502379" watchObservedRunningTime="2026-07-17 20:02:39.344796311 +0000 UTC m=+394.038772459"
Jul 17 20:02:40.251294 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:40.251265    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mfjbf"
Jul 17 20:02:40.270416 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:40.270387    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 17 20:02:40.270505 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:40.270449    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 17 20:02:41.304283 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:41.304252    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8" event={"ID":"dcfa7f00-ae9c-4e06-a14d-937beeaa6381","Type":"ContainerStarted","Data":"54d82b88ebcbfa8b04101e5a8e39ee2268c21eafe2b7e80ee139adbf8695dca0"}
Jul 17 20:02:41.304773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:41.304451    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:02:41.305901 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:41.305877    2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-mfnl8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 20:02:41.306025 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:41.305923    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8" podUID="dcfa7f00-ae9c-4e06-a14d-937beeaa6381" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 20:02:41.323144 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:41.323098    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8" podStartSLOduration=1.8248340010000001 podStartE2EDuration="5.323082283s" totalImagesPullingTime="3.498248282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:02:41.321523403 +0000 UTC m=+396.015499542" watchObservedRunningTime="2026-07-17 20:02:41.323082283 +0000 UTC m=+396.017058430"
Jul 17 20:02:42.308974 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:02:42.308936    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mfnl8"
Jul 17 20:03:10.300509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:03:10.300471    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxc4f"
Jul 17 20:03:10.303425 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:03:10.303402    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s5fvl"
Jul 17 20:04:05.520567 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.520534    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"]
Jul 17 20:04:05.522653 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.522628    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:05.525773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.525755    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 20:04:05.525885 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.525776    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x2kxv\""
Jul 17 20:04:05.526536 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.526514    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 20:04:05.552269 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.552243    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"]
Jul 17 20:04:05.605372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.605339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzc5\" (UniqueName: \"kubernetes.io/projected/b0ea90e4-b25b-41b6-b933-fdcde1b39de0-kube-api-access-wqzc5\") pod \"limitador-operator-controller-manager-859d7b78bd-2wxf8\" (UID: \"b0ea90e4-b25b-41b6-b933-fdcde1b39de0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:05.706557 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.706521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzc5\" (UniqueName: \"kubernetes.io/projected/b0ea90e4-b25b-41b6-b933-fdcde1b39de0-kube-api-access-wqzc5\") pod \"limitador-operator-controller-manager-859d7b78bd-2wxf8\" (UID: \"b0ea90e4-b25b-41b6-b933-fdcde1b39de0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:05.714698 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.714675    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 20:04:05.725822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.725794    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 20:04:05.736757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.736729    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzc5\" (UniqueName: \"kubernetes.io/projected/b0ea90e4-b25b-41b6-b933-fdcde1b39de0-kube-api-access-wqzc5\") pod \"limitador-operator-controller-manager-859d7b78bd-2wxf8\" (UID: \"b0ea90e4-b25b-41b6-b933-fdcde1b39de0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:05.835202 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.835126    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x2kxv\""
Jul 17 20:04:05.843132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:05.843106    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:06.006215 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:06.006181    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"]
Jul 17 20:04:06.009378 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:04:06.009344    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ea90e4_b25b_41b6_b933_fdcde1b39de0.slice/crio-4966c3a46f5388d33f65df5f80b74cc7b9f651f76f3d9ac7382e5665f4f518eb WatchSource:0}: Error finding container 4966c3a46f5388d33f65df5f80b74cc7b9f651f76f3d9ac7382e5665f4f518eb: Status 404 returned error can't find the container with id 4966c3a46f5388d33f65df5f80b74cc7b9f651f76f3d9ac7382e5665f4f518eb
Jul 17 20:04:06.607850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:06.607816    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8" event={"ID":"b0ea90e4-b25b-41b6-b933-fdcde1b39de0","Type":"ContainerStarted","Data":"4966c3a46f5388d33f65df5f80b74cc7b9f651f76f3d9ac7382e5665f4f518eb"}
Jul 17 20:04:08.619449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:08.619409    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8" event={"ID":"b0ea90e4-b25b-41b6-b933-fdcde1b39de0","Type":"ContainerStarted","Data":"d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1"}
Jul 17 20:04:08.619926 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:08.619542    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:08.644162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:08.644112    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8" podStartSLOduration=1.924246152 podStartE2EDuration="3.644082669s" totalImagesPullingTime="1.719836517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:04:08.643545683 +0000 UTC m=+483.337521830" watchObservedRunningTime="2026-07-17 20:04:08.644082669 +0000 UTC m=+483.338058815"
Jul 17 20:04:19.625399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:19.625365    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:29.586756 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.586718    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"]
Jul 17 20:04:29.589915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.589900    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.592410 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.592390    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jbh4k\""
Jul 17 20:04:29.605152 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.605126    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"]
Jul 17 20:04:29.632345 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.632314    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"]
Jul 17 20:04:29.632593 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.632543    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8" podUID="b0ea90e4-b25b-41b6-b933-fdcde1b39de0" containerName="manager" containerID="cri-o://d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1" gracePeriod=2
Jul 17 20:04:29.646897 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.646753    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"]
Jul 17 20:04:29.660742 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.660715    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"]
Jul 17 20:04:29.661097 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.661081    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ea90e4-b25b-41b6-b933-fdcde1b39de0" containerName="manager"
Jul 17 20:04:29.661190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.661100    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ea90e4-b25b-41b6-b933-fdcde1b39de0" containerName="manager"
Jul 17 20:04:29.661190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.661172    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ea90e4-b25b-41b6-b933-fdcde1b39de0" containerName="manager"
Jul 17 20:04:29.664064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.664029    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"
Jul 17 20:04:29.676329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.676304    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"]
Jul 17 20:04:29.689498 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:04:29.689447    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2wxf8\" is forbidden: User \"system:node:ip-10-0-143-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-185.ec2.internal' and this object" podUID="b0ea90e4-b25b-41b6-b933-fdcde1b39de0" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:29.704037 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.704006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhvt\" (UniqueName: \"kubernetes.io/projected/08ae4b01-42a3-4bcd-b34e-74112b55789a-kube-api-access-jnhvt\") pod \"kuadrant-operator-controller-manager-86878c769f-b2x75\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.704255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.704233    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ae4b01-42a3-4bcd-b34e-74112b55789a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b2x75\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.805619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.805583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ae4b01-42a3-4bcd-b34e-74112b55789a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b2x75\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.805744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.805648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8c4\" (UniqueName: \"kubernetes.io/projected/ee60f502-c989-4a07-8561-d62e6c81bb7f-kube-api-access-vd8c4\") pod \"limitador-operator-controller-manager-859d7b78bd-bj95b\" (UID: \"ee60f502-c989-4a07-8561-d62e6c81bb7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"
Jul 17 20:04:29.805744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.805724    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhvt\" (UniqueName: \"kubernetes.io/projected/08ae4b01-42a3-4bcd-b34e-74112b55789a-kube-api-access-jnhvt\") pod \"kuadrant-operator-controller-manager-86878c769f-b2x75\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.806875 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.806239    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ae4b01-42a3-4bcd-b34e-74112b55789a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b2x75\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.824014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.823978    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhvt\" (UniqueName: \"kubernetes.io/projected/08ae4b01-42a3-4bcd-b34e-74112b55789a-kube-api-access-jnhvt\") pod \"kuadrant-operator-controller-manager-86878c769f-b2x75\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.872415 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.872391    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:29.899200 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.899174    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:29.907070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.907018    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8c4\" (UniqueName: \"kubernetes.io/projected/ee60f502-c989-4a07-8561-d62e6c81bb7f-kube-api-access-vd8c4\") pod \"limitador-operator-controller-manager-859d7b78bd-bj95b\" (UID: \"ee60f502-c989-4a07-8561-d62e6c81bb7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"
Jul 17 20:04:29.915807 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:29.915773    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8c4\" (UniqueName: \"kubernetes.io/projected/ee60f502-c989-4a07-8561-d62e6c81bb7f-kube-api-access-vd8c4\") pod \"limitador-operator-controller-manager-859d7b78bd-bj95b\" (UID: \"ee60f502-c989-4a07-8561-d62e6c81bb7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"
Jul 17 20:04:30.007433 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.007403    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqzc5\" (UniqueName: \"kubernetes.io/projected/b0ea90e4-b25b-41b6-b933-fdcde1b39de0-kube-api-access-wqzc5\") pod \"b0ea90e4-b25b-41b6-b933-fdcde1b39de0\" (UID: \"b0ea90e4-b25b-41b6-b933-fdcde1b39de0\") "
Jul 17 20:04:30.009579 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.009549    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ea90e4-b25b-41b6-b933-fdcde1b39de0-kube-api-access-wqzc5" (OuterVolumeSpecName: "kube-api-access-wqzc5") pod "b0ea90e4-b25b-41b6-b933-fdcde1b39de0" (UID: "b0ea90e4-b25b-41b6-b933-fdcde1b39de0"). InnerVolumeSpecName "kube-api-access-wqzc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:04:30.011703 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.011681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"
Jul 17 20:04:30.030842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.030811    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"]
Jul 17 20:04:30.032702 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:04:30.032675    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ae4b01_42a3_4bcd_b34e_74112b55789a.slice/crio-7de99b1c49389aa1a1c9fff10804bec908703e94d4de19c05d83ca890a7d3091 WatchSource:0}: Error finding container 7de99b1c49389aa1a1c9fff10804bec908703e94d4de19c05d83ca890a7d3091: Status 404 returned error can't find the container with id 7de99b1c49389aa1a1c9fff10804bec908703e94d4de19c05d83ca890a7d3091
Jul 17 20:04:30.108738 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.108655    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzc5\" (UniqueName: \"kubernetes.io/projected/b0ea90e4-b25b-41b6-b933-fdcde1b39de0-kube-api-access-wqzc5\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:04:30.158911 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.158886    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"]
Jul 17 20:04:30.161235 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:04:30.161207    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee60f502_c989_4a07_8561_d62e6c81bb7f.slice/crio-4d242a6f1ef51fe272e815bd9f89ec69c26a79d702cc93d50684c85ba457e746 WatchSource:0}: Error finding container 4d242a6f1ef51fe272e815bd9f89ec69c26a79d702cc93d50684c85ba457e746: Status 404 returned error can't find the container with id 4d242a6f1ef51fe272e815bd9f89ec69c26a79d702cc93d50684c85ba457e746
Jul 17 20:04:30.710122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.710082    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b" event={"ID":"ee60f502-c989-4a07-8561-d62e6c81bb7f","Type":"ContainerStarted","Data":"f761fba0eed67f4bc32dfa0b8b462b2144eedf86c0dfcc6707a000e2ccdc6008"}
Jul 17 20:04:30.710596 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.710145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b" event={"ID":"ee60f502-c989-4a07-8561-d62e6c81bb7f","Type":"ContainerStarted","Data":"4d242a6f1ef51fe272e815bd9f89ec69c26a79d702cc93d50684c85ba457e746"}
Jul 17 20:04:30.710596 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.710179    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"
Jul 17 20:04:30.711652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.711629    2572 generic.go:356] "Generic (PLEG): container finished" podID="b0ea90e4-b25b-41b6-b933-fdcde1b39de0" containerID="d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1" exitCode=0
Jul 17 20:04:30.711777 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.711674    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2wxf8"
Jul 17 20:04:30.711777 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.711732    2572 scope.go:117] "RemoveContainer" containerID="d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1"
Jul 17 20:04:30.713539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.713261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75" event={"ID":"08ae4b01-42a3-4bcd-b34e-74112b55789a","Type":"ContainerStarted","Data":"7de99b1c49389aa1a1c9fff10804bec908703e94d4de19c05d83ca890a7d3091"}
Jul 17 20:04:30.727785 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.727729    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b" podStartSLOduration=1.727709812 podStartE2EDuration="1.727709812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:04:30.726242944 +0000 UTC m=+505.420219087" watchObservedRunningTime="2026-07-17 20:04:30.727709812 +0000 UTC m=+505.421685936"
Jul 17 20:04:30.800812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.800791    2572 scope.go:117] "RemoveContainer" containerID="d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1"
Jul 17 20:04:30.801236 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:04:30.801154    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1\": container with ID starting with d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1 not found: ID does not exist" containerID="d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1"
Jul 17 20:04:30.801236 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:30.801210    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1"} err="failed to get container status \"d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1\": rpc error: code = NotFound desc = could not find container \"d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1\": container with ID starting with d3f90610f017ba8a2491c69d84dc03e6b1ca27109efeea2816e433e6c5943ae1 not found: ID does not exist"
Jul 17 20:04:31.855225 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:31.855190    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ea90e4-b25b-41b6-b933-fdcde1b39de0" path="/var/lib/kubelet/pods/b0ea90e4-b25b-41b6-b933-fdcde1b39de0/volumes"
Jul 17 20:04:35.740902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:35.740871    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75" event={"ID":"08ae4b01-42a3-4bcd-b34e-74112b55789a","Type":"ContainerStarted","Data":"f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6"}
Jul 17 20:04:35.741295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:35.741004    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:04:35.762809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:35.762762    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75" podStartSLOduration=1.734302403 podStartE2EDuration="6.762746938s" totalImagesPullingTime="5.028444535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:04:35.76071452 +0000 UTC m=+510.454690658" watchObservedRunningTime="2026-07-17 20:04:35.762746938 +0000 UTC m=+510.456723084"
Jul 17 20:04:41.721692 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:41.721659    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bj95b"
Jul 17 20:04:46.747790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:04:46.747762    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:05:05.395940 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.395846    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:05:05.399780 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.399759    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.403316 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.403296    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5hf6p\""
Jul 17 20:05:05.403686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.403668    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 20:05:05.412661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.412635    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:05:05.426167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.426137    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:05:05.493947 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.493906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpb8\" (UniqueName: \"kubernetes.io/projected/52b478c2-e1c8-47e5-b742-55a254ff683c-kube-api-access-qwpb8\") pod \"limitador-limitador-54d6d988d5-csfcv\" (UID: \"52b478c2-e1c8-47e5-b742-55a254ff683c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.494167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.494016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52b478c2-e1c8-47e5-b742-55a254ff683c-config-file\") pod \"limitador-limitador-54d6d988d5-csfcv\" (UID: \"52b478c2-e1c8-47e5-b742-55a254ff683c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.595393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.595358    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52b478c2-e1c8-47e5-b742-55a254ff683c-config-file\") pod \"limitador-limitador-54d6d988d5-csfcv\" (UID: \"52b478c2-e1c8-47e5-b742-55a254ff683c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.595566 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.595408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpb8\" (UniqueName: \"kubernetes.io/projected/52b478c2-e1c8-47e5-b742-55a254ff683c-kube-api-access-qwpb8\") pod \"limitador-limitador-54d6d988d5-csfcv\" (UID: \"52b478c2-e1c8-47e5-b742-55a254ff683c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.596039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.596020    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52b478c2-e1c8-47e5-b742-55a254ff683c-config-file\") pod \"limitador-limitador-54d6d988d5-csfcv\" (UID: \"52b478c2-e1c8-47e5-b742-55a254ff683c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.603115 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.603084    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpb8\" (UniqueName: \"kubernetes.io/projected/52b478c2-e1c8-47e5-b742-55a254ff683c-kube-api-access-qwpb8\") pod \"limitador-limitador-54d6d988d5-csfcv\" (UID: \"52b478c2-e1c8-47e5-b742-55a254ff683c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.711752 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.711729    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5hf6p\""
Jul 17 20:05:05.720479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.720458    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:05.856222 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:05.856189    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:05:05.860163 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:05:05.860134    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b478c2_e1c8_47e5_b742_55a254ff683c.slice/crio-0995c571916f92993f93f60af6eea9e2d32c9ff8217788374006304ba29e2f17 WatchSource:0}: Error finding container 0995c571916f92993f93f60af6eea9e2d32c9ff8217788374006304ba29e2f17: Status 404 returned error can't find the container with id 0995c571916f92993f93f60af6eea9e2d32c9ff8217788374006304ba29e2f17
Jul 17 20:05:06.867843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:06.867797    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv" event={"ID":"52b478c2-e1c8-47e5-b742-55a254ff683c","Type":"ContainerStarted","Data":"0995c571916f92993f93f60af6eea9e2d32c9ff8217788374006304ba29e2f17"}
Jul 17 20:05:08.877867 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:08.877832    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv" event={"ID":"52b478c2-e1c8-47e5-b742-55a254ff683c","Type":"ContainerStarted","Data":"57102978866af8d9defe46c7bcfd7f96bb5715eb19c167d09e7040d4a73b2ed8"}
Jul 17 20:05:08.878285 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:08.877965    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:05:08.894135 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:08.894088    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv" podStartSLOduration=1.336092778 podStartE2EDuration="3.894073316s" totalImagesPullingTime="2.557980538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:05:08.892230125 +0000 UTC m=+543.586206271" watchObservedRunningTime="2026-07-17 20:05:08.894073316 +0000 UTC m=+543.588049507"
Jul 17 20:05:19.883177 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:05:19.883147    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-csfcv"
Jul 17 20:06:18.259761 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:06:18.259724    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:06:42.173678 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:06:42.173647    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:06:48.375721 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:06:48.375683    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:06:55.172938 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:06:55.172902    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:07:20.984075 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:07:20.984028    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:07:32.580499 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:07:32.580463    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:07:46.880854 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:07:46.880816    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:09:29.591143 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:09:29.591031    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:09:40.278674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:09:40.278635    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:09:51.279663 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:09:51.279631    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:11:33.780273 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:11:33.780235    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:11:43.355345 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:11:43.355306    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:11:56.687924 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:11:56.687885    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:12:13.290323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:12:13.290288    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:12:22.690529 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:12:22.690487    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:12:31.496953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:12:31.496868    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:12:42.878645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:12:42.878605    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:12:53.974740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:12:53.974707    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:13:03.279803 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:13:03.279760    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:13:14.887307 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:13:14.887267    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:13:25.185439 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:13:25.185402    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:13:46.989220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:13:46.989181    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:13:56.476503 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:13:56.476451    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:14:19.185186 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:14:19.185152    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:14:31.074680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:14:31.074645    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:14:45.882824 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:14:45.882787    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:14:55.480919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:14:55.480881    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:15:00.135270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:00.135237    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9"]
Jul 17 20:15:00.138916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:00.138891    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9"
Jul 17 20:15:00.140960 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:00.140935    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:15:00.148983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:00.148966    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9"
Jul 17 20:15:00.154005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:00.153981    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9"]
Jul 17 20:15:00.280013 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:00.279849    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9"]
Jul 17 20:15:00.282721 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:15:00.282693    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be5be68_aa82_462e_9ab1_59506b52b1c0.slice/crio-172900999ee7d6b542f8893af5283ab75f1c867f0e43801367a9d3b4dcd755f2 WatchSource:0}: Error finding container 172900999ee7d6b542f8893af5283ab75f1c867f0e43801367a9d3b4dcd755f2: Status 404 returned error can't find the container with id 172900999ee7d6b542f8893af5283ab75f1c867f0e43801367a9d3b4dcd755f2
Jul 17 20:15:00.284574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:00.284560    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 20:15:01.286961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:01.286926    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9" event={"ID":"1be5be68-aa82-462e-9ab1-59506b52b1c0","Type":"ContainerStarted","Data":"172900999ee7d6b542f8893af5283ab75f1c867f0e43801367a9d3b4dcd755f2"}
Jul 17 20:15:05.185121 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:05.185086    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:15:10.709751 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:15:10.709709    2572 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:9.7: reading manifest 9.7 in registry.redhat.io/ubi9/ubi-minimal: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="registry.redhat.io/ubi9/ubi-minimal:9.7"
Jul 17 20:15:10.710154 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:15:10.709822    2572 kuberuntime_manager.go:1459] "Unhandled Error" err="container cleanup start failed in pod maas-api-key-cleanup-29738655-5zrj9_odh-ai-gateway-infra(1be5be68-aa82-462e-9ab1-59506b52b1c0): ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:9.7: reading manifest 9.7 in registry.redhat.io/ubi9/ubi-minimal: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 17 20:15:10.710154 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:15:10.709852    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:9.7: reading manifest 9.7 in registry.redhat.io/ubi9/ubi-minimal: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9" podUID="1be5be68-aa82-462e-9ab1-59506b52b1c0"
Jul 17 20:15:11.328669 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:15:11.328628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:9.7: reading manifest 9.7 in registry.redhat.io/ubi9/ubi-minimal: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9" podUID="1be5be68-aa82-462e-9ab1-59506b52b1c0"
Jul 17 20:15:14.173999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:14.173966    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:15:23.481867 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:23.481828    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:15:25.393708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:25.393678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9" event={"ID":"1be5be68-aa82-462e-9ab1-59506b52b1c0","Type":"ContainerStarted","Data":"84d6e9a68207616ff528327b51e6a9a36f12636c7cef61768257237a2ea1a61a"}
Jul 17 20:15:25.407993 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:25.407946    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9" podStartSLOduration=0.518259796 podStartE2EDuration="25.40793098s" totalImagesPullingTime="24.889671184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-17 20:15:25.406368928 +0000 UTC m=+1160.100345073" watchObservedRunningTime="2026-07-17 20:15:25.40793098 +0000 UTC m=+1160.101907138"
Jul 17 20:15:26.400208 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:26.400109    2572 generic.go:356] "Generic (PLEG): container finished" podID="1be5be68-aa82-462e-9ab1-59506b52b1c0" containerID="84d6e9a68207616ff528327b51e6a9a36f12636c7cef61768257237a2ea1a61a" exitCode=0
Jul 17 20:15:26.400208 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:26.400186    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9" event={"ID":"1be5be68-aa82-462e-9ab1-59506b52b1c0","Type":"ContainerDied","Data":"84d6e9a68207616ff528327b51e6a9a36f12636c7cef61768257237a2ea1a61a"}
Jul 17 20:15:27.538556 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:27.538526    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9"
Jul 17 20:15:28.409449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:28.409419    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9"
Jul 17 20:15:28.409449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:28.409431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738655-5zrj9" event={"ID":"1be5be68-aa82-462e-9ab1-59506b52b1c0","Type":"ContainerDied","Data":"172900999ee7d6b542f8893af5283ab75f1c867f0e43801367a9d3b4dcd755f2"}
Jul 17 20:15:28.409656 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:28.409477    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="172900999ee7d6b542f8893af5283ab75f1c867f0e43801367a9d3b4dcd755f2"
Jul 17 20:15:50.970314 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:50.970277    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"]
Jul 17 20:15:50.970710 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:50.970466    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75" podUID="08ae4b01-42a3-4bcd-b34e-74112b55789a" containerName="manager" containerID="cri-o://f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6" gracePeriod=10
Jul 17 20:15:53.932201 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:53.932176    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:15:54.047339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.047256    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnhvt\" (UniqueName: \"kubernetes.io/projected/08ae4b01-42a3-4bcd-b34e-74112b55789a-kube-api-access-jnhvt\") pod \"08ae4b01-42a3-4bcd-b34e-74112b55789a\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") "
Jul 17 20:15:54.047339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.047291    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ae4b01-42a3-4bcd-b34e-74112b55789a-extensions-socket-volume\") pod \"08ae4b01-42a3-4bcd-b34e-74112b55789a\" (UID: \"08ae4b01-42a3-4bcd-b34e-74112b55789a\") "
Jul 17 20:15:54.047672 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.047646    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ae4b01-42a3-4bcd-b34e-74112b55789a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "08ae4b01-42a3-4bcd-b34e-74112b55789a" (UID: "08ae4b01-42a3-4bcd-b34e-74112b55789a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:15:54.049178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.049155    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ae4b01-42a3-4bcd-b34e-74112b55789a-kube-api-access-jnhvt" (OuterVolumeSpecName: "kube-api-access-jnhvt") pod "08ae4b01-42a3-4bcd-b34e-74112b55789a" (UID: "08ae4b01-42a3-4bcd-b34e-74112b55789a"). InnerVolumeSpecName "kube-api-access-jnhvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:15:54.147767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.147727    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnhvt\" (UniqueName: \"kubernetes.io/projected/08ae4b01-42a3-4bcd-b34e-74112b55789a-kube-api-access-jnhvt\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:15:54.147767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.147761    2572 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ae4b01-42a3-4bcd-b34e-74112b55789a-extensions-socket-volume\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:15:54.517755 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.517720    2572 generic.go:356] "Generic (PLEG): container finished" podID="08ae4b01-42a3-4bcd-b34e-74112b55789a" containerID="f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6" exitCode=0
Jul 17 20:15:54.517954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.517765    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"
Jul 17 20:15:54.517954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.517809    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75" event={"ID":"08ae4b01-42a3-4bcd-b34e-74112b55789a","Type":"ContainerDied","Data":"f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6"}
Jul 17 20:15:54.517954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.517865    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75" event={"ID":"08ae4b01-42a3-4bcd-b34e-74112b55789a","Type":"ContainerDied","Data":"7de99b1c49389aa1a1c9fff10804bec908703e94d4de19c05d83ca890a7d3091"}
Jul 17 20:15:54.517954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.517895    2572 scope.go:117] "RemoveContainer" containerID="f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6"
Jul 17 20:15:54.526902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.526887    2572 scope.go:117] "RemoveContainer" containerID="f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6"
Jul 17 20:15:54.527212 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:15:54.527193    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6\": container with ID starting with f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6 not found: ID does not exist" containerID="f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6"
Jul 17 20:15:54.527304 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.527222    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6"} err="failed to get container status \"f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6\": rpc error: code = NotFound desc = could not find container \"f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6\": container with ID starting with f02940d7923399fe9df67e9f980d3f20ece4dc34fe1066b95be16026924097f6 not found: ID does not exist"
Jul 17 20:15:54.537602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.537582    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"]
Jul 17 20:15:54.541230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:54.541211    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2x75"]
Jul 17 20:15:55.853370 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:15:55.853338    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ae4b01-42a3-4bcd-b34e-74112b55789a" path="/var/lib/kubelet/pods/08ae4b01-42a3-4bcd-b34e-74112b55789a/volumes"
Jul 17 20:16:52.079281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.079245    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"]
Jul 17 20:16:52.081420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.079526    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ae4b01-42a3-4bcd-b34e-74112b55789a" containerName="manager"
Jul 17 20:16:52.081420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.079535    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ae4b01-42a3-4bcd-b34e-74112b55789a" containerName="manager"
Jul 17 20:16:52.081420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.079554    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1be5be68-aa82-462e-9ab1-59506b52b1c0" containerName="cleanup"
Jul 17 20:16:52.081420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.079560    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1be5be68-aa82-462e-9ab1-59506b52b1c0" containerName="cleanup"
Jul 17 20:16:52.081420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.079602    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1be5be68-aa82-462e-9ab1-59506b52b1c0" containerName="cleanup"
Jul 17 20:16:52.081420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.079609    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ae4b01-42a3-4bcd-b34e-74112b55789a" containerName="manager"
Jul 17 20:16:52.082210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.082195    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.084437 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.084412    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jbh4k\""
Jul 17 20:16:52.095267 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.095244    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"]
Jul 17 20:16:52.187724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.187686    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9p8\" (UniqueName: \"kubernetes.io/projected/87dbe9f7-be3a-45d4-b8a9-0437ab606d7a-kube-api-access-vd9p8\") pod \"kuadrant-operator-controller-manager-86878c769f-6xkvq\" (UID: \"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.187896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.187733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dbe9f7-be3a-45d4-b8a9-0437ab606d7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6xkvq\" (UID: \"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.289144 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.289111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9p8\" (UniqueName: \"kubernetes.io/projected/87dbe9f7-be3a-45d4-b8a9-0437ab606d7a-kube-api-access-vd9p8\") pod \"kuadrant-operator-controller-manager-86878c769f-6xkvq\" (UID: \"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.289325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.289170    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dbe9f7-be3a-45d4-b8a9-0437ab606d7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6xkvq\" (UID: \"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.289540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.289520    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dbe9f7-be3a-45d4-b8a9-0437ab606d7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6xkvq\" (UID: \"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.299607 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.299586    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9p8\" (UniqueName: \"kubernetes.io/projected/87dbe9f7-be3a-45d4-b8a9-0437ab606d7a-kube-api-access-vd9p8\") pod \"kuadrant-operator-controller-manager-86878c769f-6xkvq\" (UID: \"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.391440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.391361    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.525640 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.525613    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"]
Jul 17 20:16:52.528132 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:16:52.528107    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dbe9f7_be3a_45d4_b8a9_0437ab606d7a.slice/crio-7132c02cc7da86d9e1a7f20207c70b13b245a0f2436012f987c1ace390c00165 WatchSource:0}: Error finding container 7132c02cc7da86d9e1a7f20207c70b13b245a0f2436012f987c1ace390c00165: Status 404 returned error can't find the container with id 7132c02cc7da86d9e1a7f20207c70b13b245a0f2436012f987c1ace390c00165
Jul 17 20:16:52.756026 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.755993    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq" event={"ID":"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a","Type":"ContainerStarted","Data":"57dc240cd3b92fb368bcb7957d94db7fa8eea48e11dc44330f5bc0bff81d7a5a"}
Jul 17 20:16:52.756207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.756041    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq" event={"ID":"87dbe9f7-be3a-45d4-b8a9-0437ab606d7a","Type":"ContainerStarted","Data":"7132c02cc7da86d9e1a7f20207c70b13b245a0f2436012f987c1ace390c00165"}
Jul 17 20:16:52.756207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.756086    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:16:52.775574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:16:52.775336    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq" podStartSLOduration=0.77532075 podStartE2EDuration="775.32075ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:16:52.774225156 +0000 UTC m=+1247.468201318" watchObservedRunningTime="2026-07-17 20:16:52.77532075 +0000 UTC m=+1247.469296894"
Jul 17 20:17:03.762447 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:17:03.762372    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6xkvq"
Jul 17 20:17:13.076914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:17:13.076869    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:17:22.476888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:17:22.476843    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:17:34.976488 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:17:34.976452    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:17:46.579839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:17:46.579800    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:17:58.399853 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:17:58.399818    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:18:07.682888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:18:07.682852    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:18:16.579547 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:18:16.579509    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:18:28.179403 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:18:28.179326    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:18:37.179964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:18:37.179927    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:18:47.684307 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:18:47.684265    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:18:56.683348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:18:56.683318    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:19:19.478868 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:19:19.478831    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:19:28.677858 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:19:28.677820    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:19:38.285093 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:19:38.285059    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:19:50.674011 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:19:50.673975    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:20:06.480241 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:20:06.480157    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:20:15.872832 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:20:15.872795    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:20:24.475726 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:20:24.475687    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:20:34.985883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:20:34.985843    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:20:44.385506 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:20:44.385462    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:20:54.079031 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:20:54.078996    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:21:07.379761 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:21:07.379729    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:21:16.384572 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:21:16.384536    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:21:26.079685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:21:26.079654    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:21:35.675759 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:21:35.675677    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:21:48.383966 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:21:48.383923    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:21:57.481780 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:21:57.481743    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:22:09.681668 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:22:09.681631    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:22:20.278137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:22:20.278097    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:22:31.975240 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:22:31.975198    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:22:42.678436 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:22:42.678400    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:22:59.678498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:22:59.678407    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:23:08.884003 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:08.883968    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:23:10.301993 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.301958    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"]
Jul 17 20:23:10.305003 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.304986    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"
Jul 17 20:23:10.307000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.306979    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:23:10.307125 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.306979    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:23:10.312023 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.312000    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"]
Jul 17 20:23:10.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.396702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7dc9d5-6d82-4b82-9620-b89695ab7b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-ncf72\" (UID: \"da7dc9d5-6d82-4b82-9620-b89695ab7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"
Jul 17 20:23:10.497538 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.497502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7dc9d5-6d82-4b82-9620-b89695ab7b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-ncf72\" (UID: \"da7dc9d5-6d82-4b82-9620-b89695ab7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"
Jul 17 20:23:10.498163 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.498141    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7dc9d5-6d82-4b82-9620-b89695ab7b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-ncf72\" (UID: \"da7dc9d5-6d82-4b82-9620-b89695ab7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"
Jul 17 20:23:10.616913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.616815    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"
Jul 17 20:23:10.738014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:10.737982    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"]
Jul 17 20:23:10.740796 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:23:10.740766    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7dc9d5_6d82_4b82_9620_b89695ab7b59.slice/crio-4d4cc16eda2f74bc47e3cd0f0bad887b71b3071f97440e070561584eaaba060a WatchSource:0}: Error finding container 4d4cc16eda2f74bc47e3cd0f0bad887b71b3071f97440e070561584eaaba060a: Status 404 returned error can't find the container with id 4d4cc16eda2f74bc47e3cd0f0bad887b71b3071f97440e070561584eaaba060a
Jul 17 20:23:11.337600 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:11.337573    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" event={"ID":"da7dc9d5-6d82-4b82-9620-b89695ab7b59","Type":"ContainerStarted","Data":"cc8ce63ad1d4e62926267e3195d5669ce936c1382c5ec15cba8a41c39c3957c2"}
Jul 17 20:23:11.337883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:11.337655    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" event={"ID":"da7dc9d5-6d82-4b82-9620-b89695ab7b59","Type":"ContainerStarted","Data":"4d4cc16eda2f74bc47e3cd0f0bad887b71b3071f97440e070561584eaaba060a"}
Jul 17 20:23:11.352286 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:11.352243    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" podStartSLOduration=1.3522279830000001 podStartE2EDuration="1.352227983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:23:11.350479227 +0000 UTC m=+1626.044455356" watchObservedRunningTime="2026-07-17 20:23:11.352227983 +0000 UTC m=+1626.046204156"
Jul 17 20:23:12.344102 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:12.344068    2572 generic.go:356] "Generic (PLEG): container finished" podID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerID="cc8ce63ad1d4e62926267e3195d5669ce936c1382c5ec15cba8a41c39c3957c2" exitCode=7
Jul 17 20:23:12.344486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:12.344146    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" event={"ID":"da7dc9d5-6d82-4b82-9620-b89695ab7b59","Type":"ContainerDied","Data":"cc8ce63ad1d4e62926267e3195d5669ce936c1382c5ec15cba8a41c39c3957c2"}
Jul 17 20:23:12.344486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:12.344321    2572 scope.go:117] "RemoveContainer" containerID="cc8ce63ad1d4e62926267e3195d5669ce936c1382c5ec15cba8a41c39c3957c2"
Jul 17 20:23:13.349486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:13.349448    2572 generic.go:356] "Generic (PLEG): container finished" podID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerID="e2e7327566037f1bf57b6798e725536a9ccc7ccf6fce984f5f420bfc84e84bad" exitCode=7
Jul 17 20:23:13.349942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:13.349539    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" event={"ID":"da7dc9d5-6d82-4b82-9620-b89695ab7b59","Type":"ContainerDied","Data":"e2e7327566037f1bf57b6798e725536a9ccc7ccf6fce984f5f420bfc84e84bad"}
Jul 17 20:23:13.349942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:13.349613    2572 scope.go:117] "RemoveContainer" containerID="cc8ce63ad1d4e62926267e3195d5669ce936c1382c5ec15cba8a41c39c3957c2"
Jul 17 20:23:13.349942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:13.349797    2572 scope.go:117] "RemoveContainer" containerID="e2e7327566037f1bf57b6798e725536a9ccc7ccf6fce984f5f420bfc84e84bad"
Jul 17 20:23:13.350125 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:23:13.349971    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-ncf72_odh-ai-gateway-infra(da7dc9d5-6d82-4b82-9620-b89695ab7b59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59"
Jul 17 20:23:14.355429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:14.355402    2572 scope.go:117] "RemoveContainer" containerID="e2e7327566037f1bf57b6798e725536a9ccc7ccf6fce984f5f420bfc84e84bad"
Jul 17 20:23:14.355790 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:23:14.355567    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-ncf72_odh-ai-gateway-infra(da7dc9d5-6d82-4b82-9620-b89695ab7b59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59"
Jul 17 20:23:28.848342 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:28.848309    2572 scope.go:117] "RemoveContainer" containerID="e2e7327566037f1bf57b6798e725536a9ccc7ccf6fce984f5f420bfc84e84bad"
Jul 17 20:23:29.420593 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:29.420559    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" event={"ID":"da7dc9d5-6d82-4b82-9620-b89695ab7b59","Type":"ContainerStarted","Data":"f62274a0c42186d4c734d62ca111bb537a251e916ad08b7fe3782217032f1a18"}
Jul 17 20:23:30.426582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:30.426545    2572 generic.go:356] "Generic (PLEG): container finished" podID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerID="f62274a0c42186d4c734d62ca111bb537a251e916ad08b7fe3782217032f1a18" exitCode=7
Jul 17 20:23:30.427072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:30.426593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" event={"ID":"da7dc9d5-6d82-4b82-9620-b89695ab7b59","Type":"ContainerDied","Data":"f62274a0c42186d4c734d62ca111bb537a251e916ad08b7fe3782217032f1a18"}
Jul 17 20:23:30.427072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:30.426651    2572 scope.go:117] "RemoveContainer" containerID="e2e7327566037f1bf57b6798e725536a9ccc7ccf6fce984f5f420bfc84e84bad"
Jul 17 20:23:30.427072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:30.426933    2572 scope.go:117] "RemoveContainer" containerID="f62274a0c42186d4c734d62ca111bb537a251e916ad08b7fe3782217032f1a18"
Jul 17 20:23:30.427194 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:23:30.427172    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-ncf72_odh-ai-gateway-infra(da7dc9d5-6d82-4b82-9620-b89695ab7b59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59"
Jul 17 20:23:30.450172 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:30.450133    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"]
Jul 17 20:23:31.563608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:31.563578    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"
Jul 17 20:23:31.667986 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:31.667949    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7dc9d5-6d82-4b82-9620-b89695ab7b59-maas-api-service-ca\") pod \"da7dc9d5-6d82-4b82-9620-b89695ab7b59\" (UID: \"da7dc9d5-6d82-4b82-9620-b89695ab7b59\") "
Jul 17 20:23:31.668417 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:31.668388    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7dc9d5-6d82-4b82-9620-b89695ab7b59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da7dc9d5-6d82-4b82-9620-b89695ab7b59" (UID: "da7dc9d5-6d82-4b82-9620-b89695ab7b59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:23:31.769377 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:31.769335    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7dc9d5-6d82-4b82-9620-b89695ab7b59-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:23:32.437271 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:32.437243    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"
Jul 17 20:23:32.437271 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:32.437258    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72" event={"ID":"da7dc9d5-6d82-4b82-9620-b89695ab7b59","Type":"ContainerDied","Data":"4d4cc16eda2f74bc47e3cd0f0bad887b71b3071f97440e070561584eaaba060a"}
Jul 17 20:23:32.437490 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:32.437313    2572 scope.go:117] "RemoveContainer" containerID="f62274a0c42186d4c734d62ca111bb537a251e916ad08b7fe3782217032f1a18"
Jul 17 20:23:32.450952 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:32.450922    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"]
Jul 17 20:23:32.457261 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:32.457235    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-ncf72"]
Jul 17 20:23:33.858088 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:23:33.858038    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" path="/var/lib/kubelet/pods/da7dc9d5-6d82-4b82-9620-b89695ab7b59/volumes"
Jul 17 20:24:10.348668 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.348634    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"]
Jul 17 20:24:10.349081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.348919    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:24:10.349081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.348929    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:24:10.349081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.348938    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:24:10.349081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.348944    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:24:10.349081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.349000    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:24:10.349081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.349009    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:24:10.351753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.351739    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"
Jul 17 20:24:10.353782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.353762    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:24:10.353782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.353777    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:24:10.359260 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.359237    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"]
Jul 17 20:24:10.475987 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.475951    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed45b33-2025-492d-aec1-d0074abf18bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-g228h\" (UID: \"9ed45b33-2025-492d-aec1-d0074abf18bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"
Jul 17 20:24:10.577322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.577284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed45b33-2025-492d-aec1-d0074abf18bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-g228h\" (UID: \"9ed45b33-2025-492d-aec1-d0074abf18bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"
Jul 17 20:24:10.577920 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.577899    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed45b33-2025-492d-aec1-d0074abf18bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-g228h\" (UID: \"9ed45b33-2025-492d-aec1-d0074abf18bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"
Jul 17 20:24:10.662534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.662446    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"
Jul 17 20:24:10.788346 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:10.788323    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"]
Jul 17 20:24:10.790528 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:24:10.790499    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed45b33_2025_492d_aec1_d0074abf18bc.slice/crio-056a0cba881c832e4a92a87b5bf720e33ce9e11975d6ffed09ac6b28b4314017 WatchSource:0}: Error finding container 056a0cba881c832e4a92a87b5bf720e33ce9e11975d6ffed09ac6b28b4314017: Status 404 returned error can't find the container with id 056a0cba881c832e4a92a87b5bf720e33ce9e11975d6ffed09ac6b28b4314017
Jul 17 20:24:11.608661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:11.608629    2572 generic.go:356] "Generic (PLEG): container finished" podID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerID="0c55d967dc3565c383e1372811ebd1326874a5e1b6d8c2a8ed796f72171704f7" exitCode=7
Jul 17 20:24:11.609100 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:11.608697    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" event={"ID":"9ed45b33-2025-492d-aec1-d0074abf18bc","Type":"ContainerDied","Data":"0c55d967dc3565c383e1372811ebd1326874a5e1b6d8c2a8ed796f72171704f7"}
Jul 17 20:24:11.609100 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:11.608739    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" event={"ID":"9ed45b33-2025-492d-aec1-d0074abf18bc","Type":"ContainerStarted","Data":"056a0cba881c832e4a92a87b5bf720e33ce9e11975d6ffed09ac6b28b4314017"}
Jul 17 20:24:11.609100 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:11.608886    2572 scope.go:117] "RemoveContainer" containerID="0c55d967dc3565c383e1372811ebd1326874a5e1b6d8c2a8ed796f72171704f7"
Jul 17 20:24:12.614953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:12.614867    2572 generic.go:356] "Generic (PLEG): container finished" podID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerID="85e6992f7cd4b863b31b3a55be9e910e46d1ae05d8cad0288ddec9482b244669" exitCode=7
Jul 17 20:24:12.614953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:12.614912    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" event={"ID":"9ed45b33-2025-492d-aec1-d0074abf18bc","Type":"ContainerDied","Data":"85e6992f7cd4b863b31b3a55be9e910e46d1ae05d8cad0288ddec9482b244669"}
Jul 17 20:24:12.615463 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:12.614962    2572 scope.go:117] "RemoveContainer" containerID="0c55d967dc3565c383e1372811ebd1326874a5e1b6d8c2a8ed796f72171704f7"
Jul 17 20:24:12.615463 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:12.615151    2572 scope.go:117] "RemoveContainer" containerID="85e6992f7cd4b863b31b3a55be9e910e46d1ae05d8cad0288ddec9482b244669"
Jul 17 20:24:12.615463 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:24:12.615339    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-g228h_odh-ai-gateway-infra(9ed45b33-2025-492d-aec1-d0074abf18bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc"
Jul 17 20:24:13.620191 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:13.620167    2572 scope.go:117] "RemoveContainer" containerID="85e6992f7cd4b863b31b3a55be9e910e46d1ae05d8cad0288ddec9482b244669"
Jul 17 20:24:13.620550 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:24:13.620326    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-g228h_odh-ai-gateway-infra(9ed45b33-2025-492d-aec1-d0074abf18bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc"
Jul 17 20:24:24.848371 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:24.848336    2572 scope.go:117] "RemoveContainer" containerID="85e6992f7cd4b863b31b3a55be9e910e46d1ae05d8cad0288ddec9482b244669"
Jul 17 20:24:25.669080 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:25.669030    2572 generic.go:356] "Generic (PLEG): container finished" podID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerID="817c650987173d4f2eea22d9302c23a42765ed5c6821e7ffb69c08c54f9f9db9" exitCode=7
Jul 17 20:24:25.669241 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:25.669097    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" event={"ID":"9ed45b33-2025-492d-aec1-d0074abf18bc","Type":"ContainerDied","Data":"817c650987173d4f2eea22d9302c23a42765ed5c6821e7ffb69c08c54f9f9db9"}
Jul 17 20:24:25.669241 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:25.669155    2572 scope.go:117] "RemoveContainer" containerID="85e6992f7cd4b863b31b3a55be9e910e46d1ae05d8cad0288ddec9482b244669"
Jul 17 20:24:25.669439 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:25.669423    2572 scope.go:117] "RemoveContainer" containerID="817c650987173d4f2eea22d9302c23a42765ed5c6821e7ffb69c08c54f9f9db9"
Jul 17 20:24:25.669632 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:24:25.669613    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-g228h_odh-ai-gateway-infra(9ed45b33-2025-492d-aec1-d0074abf18bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc"
Jul 17 20:24:26.695669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:26.695634    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"]
Jul 17 20:24:26.831356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:26.831333    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"
Jul 17 20:24:26.902629 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:26.902598    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed45b33-2025-492d-aec1-d0074abf18bc-maas-api-service-ca\") pod \"9ed45b33-2025-492d-aec1-d0074abf18bc\" (UID: \"9ed45b33-2025-492d-aec1-d0074abf18bc\") "
Jul 17 20:24:26.903004 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:26.902977    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed45b33-2025-492d-aec1-d0074abf18bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ed45b33-2025-492d-aec1-d0074abf18bc" (UID: "9ed45b33-2025-492d-aec1-d0074abf18bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:24:27.003791 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:27.003762    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed45b33-2025-492d-aec1-d0074abf18bc-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:24:27.679321 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:27.679299    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"
Jul 17 20:24:27.679507 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:27.679295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h" event={"ID":"9ed45b33-2025-492d-aec1-d0074abf18bc","Type":"ContainerDied","Data":"056a0cba881c832e4a92a87b5bf720e33ce9e11975d6ffed09ac6b28b4314017"}
Jul 17 20:24:27.679507 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:27.679451    2572 scope.go:117] "RemoveContainer" containerID="817c650987173d4f2eea22d9302c23a42765ed5c6821e7ffb69c08c54f9f9db9"
Jul 17 20:24:27.699856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:27.699829    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"]
Jul 17 20:24:27.707029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:27.707004    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-g228h"]
Jul 17 20:24:27.853641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:24:27.853605    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" path="/var/lib/kubelet/pods/9ed45b33-2025-492d-aec1-d0074abf18bc/volumes"
Jul 17 20:25:00.396275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396239    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"]
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396526    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396537    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396555    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396561    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396572    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396578    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396626    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7dc9d5-6d82-4b82-9620-b89695ab7b59" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396634    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:25:00.396732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.396640    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:25:00.399543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.399525    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"
Jul 17 20:25:00.401769 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.401748    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:25:00.401867 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.401849    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:25:00.406146 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.406124    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"]
Jul 17 20:25:00.466140 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.466102    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-t6zrq\" (UID: \"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"
Jul 17 20:25:00.567173 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.567136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-t6zrq\" (UID: \"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"
Jul 17 20:25:00.567770 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.567748    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-t6zrq\" (UID: \"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"
Jul 17 20:25:00.710598 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.710575    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"
Jul 17 20:25:00.834092 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:00.834067    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"]
Jul 17 20:25:00.836571 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:25:00.836544    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1822bd5_6f65_44a4_8ed9_9ddf9f47a90b.slice/crio-f6958310681d484d34c7dca5389477d859ece34ec06213f830b236c1f6578400 WatchSource:0}: Error finding container f6958310681d484d34c7dca5389477d859ece34ec06213f830b236c1f6578400: Status 404 returned error can't find the container with id f6958310681d484d34c7dca5389477d859ece34ec06213f830b236c1f6578400
Jul 17 20:25:01.821821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:01.821785    2572 generic.go:356] "Generic (PLEG): container finished" podID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerID="85d46b84dc04f94a53160231db698e783d4b3629a15170cb54ced8a306170846" exitCode=7
Jul 17 20:25:01.822210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:01.821830    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" event={"ID":"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b","Type":"ContainerDied","Data":"85d46b84dc04f94a53160231db698e783d4b3629a15170cb54ced8a306170846"}
Jul 17 20:25:01.822210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:01.821876    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" event={"ID":"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b","Type":"ContainerStarted","Data":"f6958310681d484d34c7dca5389477d859ece34ec06213f830b236c1f6578400"}
Jul 17 20:25:01.822210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:01.822091    2572 scope.go:117] "RemoveContainer" containerID="85d46b84dc04f94a53160231db698e783d4b3629a15170cb54ced8a306170846"
Jul 17 20:25:02.827583 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:02.827492    2572 generic.go:356] "Generic (PLEG): container finished" podID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerID="ca04c23f513eca225c360ba0b1ae929b8e00c18bc2ac643e8556d2fb97438917" exitCode=7
Jul 17 20:25:02.827927 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:02.827582    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" event={"ID":"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b","Type":"ContainerDied","Data":"ca04c23f513eca225c360ba0b1ae929b8e00c18bc2ac643e8556d2fb97438917"}
Jul 17 20:25:02.827927 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:02.827687    2572 scope.go:117] "RemoveContainer" containerID="85d46b84dc04f94a53160231db698e783d4b3629a15170cb54ced8a306170846"
Jul 17 20:25:02.827927 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:02.827847    2572 scope.go:117] "RemoveContainer" containerID="ca04c23f513eca225c360ba0b1ae929b8e00c18bc2ac643e8556d2fb97438917"
Jul 17 20:25:02.828082 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:25:02.828064    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-t6zrq_odh-ai-gateway-infra(f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b"
Jul 17 20:25:03.833634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:03.833612    2572 scope.go:117] "RemoveContainer" containerID="ca04c23f513eca225c360ba0b1ae929b8e00c18bc2ac643e8556d2fb97438917"
Jul 17 20:25:03.833968 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:25:03.833790    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-t6zrq_odh-ai-gateway-infra(f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b"
Jul 17 20:25:17.848497 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:17.848462    2572 scope.go:117] "RemoveContainer" containerID="ca04c23f513eca225c360ba0b1ae929b8e00c18bc2ac643e8556d2fb97438917"
Jul 17 20:25:18.893545 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:18.893512    2572 generic.go:356] "Generic (PLEG): container finished" podID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerID="a51ff52745da6853ad65263351dd0c3699734882b08d796266d55b123cdbf2ce" exitCode=7
Jul 17 20:25:18.893942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:18.893586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" event={"ID":"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b","Type":"ContainerDied","Data":"a51ff52745da6853ad65263351dd0c3699734882b08d796266d55b123cdbf2ce"}
Jul 17 20:25:18.893942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:18.893642    2572 scope.go:117] "RemoveContainer" containerID="ca04c23f513eca225c360ba0b1ae929b8e00c18bc2ac643e8556d2fb97438917"
Jul 17 20:25:18.893942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:18.893896    2572 scope.go:117] "RemoveContainer" containerID="a51ff52745da6853ad65263351dd0c3699734882b08d796266d55b123cdbf2ce"
Jul 17 20:25:18.894094 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:25:18.894072    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-t6zrq_odh-ai-gateway-infra(f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b"
Jul 17 20:25:19.918244 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:19.918212    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"]
Jul 17 20:25:20.050306 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.050286    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"
Jul 17 20:25:20.226489 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.226460    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b-maas-api-service-ca\") pod \"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b\" (UID: \"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b\") "
Jul 17 20:25:20.226821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.226797    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" (UID: "f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:25:20.326980 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.326948    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:25:20.903800 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.903773    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"
Jul 17 20:25:20.903800 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.903786    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq" event={"ID":"f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b","Type":"ContainerDied","Data":"f6958310681d484d34c7dca5389477d859ece34ec06213f830b236c1f6578400"}
Jul 17 20:25:20.904023 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.903837    2572 scope.go:117] "RemoveContainer" containerID="a51ff52745da6853ad65263351dd0c3699734882b08d796266d55b123cdbf2ce"
Jul 17 20:25:20.924540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.924514    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"]
Jul 17 20:25:20.930119 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:20.930092    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-t6zrq"]
Jul 17 20:25:21.853292 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:25:21.853259    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" path="/var/lib/kubelet/pods/f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b/volumes"
Jul 17 20:26:00.463578 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463495    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"]
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463793    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463803    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463821    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463826    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463836    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463842    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463888    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463895    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:00.464022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.463901    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed45b33-2025-492d-aec1-d0074abf18bc" containerName="revoke-keys"
Jul 17 20:26:00.466643 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.466620    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"
Jul 17 20:26:00.469239 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.469217    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:26:00.469358 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.469247    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:26:00.475040 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.475015    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"]
Jul 17 20:26:00.543513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.543473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c4b868-a2a7-4b6c-91c2-105ec7ce57fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-jdxxh\" (UID: \"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"
Jul 17 20:26:00.644357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.644319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c4b868-a2a7-4b6c-91c2-105ec7ce57fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-jdxxh\" (UID: \"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"
Jul 17 20:26:00.644922 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.644902    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c4b868-a2a7-4b6c-91c2-105ec7ce57fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-jdxxh\" (UID: \"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"
Jul 17 20:26:00.778201 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.778112    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"
Jul 17 20:26:00.905813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:00.905790    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"]
Jul 17 20:26:00.908087 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:26:00.908057    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c4b868_a2a7_4b6c_91c2_105ec7ce57fd.slice/crio-7ed6ce6f6eb966359efc24824aa4574c29b94db5ea892828209a18c08110139e WatchSource:0}: Error finding container 7ed6ce6f6eb966359efc24824aa4574c29b94db5ea892828209a18c08110139e: Status 404 returned error can't find the container with id 7ed6ce6f6eb966359efc24824aa4574c29b94db5ea892828209a18c08110139e
Jul 17 20:26:01.069007 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:01.068920    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" event={"ID":"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd","Type":"ContainerStarted","Data":"7ed6ce6f6eb966359efc24824aa4574c29b94db5ea892828209a18c08110139e"}
Jul 17 20:26:02.074017 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:02.073979    2572 generic.go:356] "Generic (PLEG): container finished" podID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerID="b6198accbfa93ba77ee00a5e1c91bc928cb9cc9d5a02b8ac8df1fb7dbc2c5420" exitCode=7
Jul 17 20:26:02.074452 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:02.074087    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" event={"ID":"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd","Type":"ContainerDied","Data":"b6198accbfa93ba77ee00a5e1c91bc928cb9cc9d5a02b8ac8df1fb7dbc2c5420"}
Jul 17 20:26:02.074452 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:02.074246    2572 scope.go:117] "RemoveContainer" containerID="b6198accbfa93ba77ee00a5e1c91bc928cb9cc9d5a02b8ac8df1fb7dbc2c5420"
Jul 17 20:26:03.080147 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:03.080068    2572 generic.go:356] "Generic (PLEG): container finished" podID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerID="ad72679bef601e86445a57ee7b53e2c3038a8bb6898da12a967647a7c5bdea2c" exitCode=7
Jul 17 20:26:03.080593 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:03.080162    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" event={"ID":"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd","Type":"ContainerDied","Data":"ad72679bef601e86445a57ee7b53e2c3038a8bb6898da12a967647a7c5bdea2c"}
Jul 17 20:26:03.080593 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:03.080223    2572 scope.go:117] "RemoveContainer" containerID="b6198accbfa93ba77ee00a5e1c91bc928cb9cc9d5a02b8ac8df1fb7dbc2c5420"
Jul 17 20:26:03.080593 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:03.080447    2572 scope.go:117] "RemoveContainer" containerID="ad72679bef601e86445a57ee7b53e2c3038a8bb6898da12a967647a7c5bdea2c"
Jul 17 20:26:03.080747 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:03.080680    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-jdxxh_odh-ai-gateway-infra(88c4b868-a2a7-4b6c-91c2-105ec7ce57fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd"
Jul 17 20:26:04.085973 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:04.085943    2572 scope.go:117] "RemoveContainer" containerID="ad72679bef601e86445a57ee7b53e2c3038a8bb6898da12a967647a7c5bdea2c"
Jul 17 20:26:04.086463 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:04.086174    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-jdxxh_odh-ai-gateway-infra(88c4b868-a2a7-4b6c-91c2-105ec7ce57fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd"
Jul 17 20:26:13.380009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:13.379973    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:26:16.848956 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:16.848924    2572 scope.go:117] "RemoveContainer" containerID="ad72679bef601e86445a57ee7b53e2c3038a8bb6898da12a967647a7c5bdea2c"
Jul 17 20:26:17.138490 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:17.138402    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" event={"ID":"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd","Type":"ContainerStarted","Data":"b7adadec4957aa7baf974191348b7d9e49e4bd2471240e477b739191d92fc1cb"}
Jul 17 20:26:17.155170 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:17.155126    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" podStartSLOduration=17.155111662 podStartE2EDuration="17.155111662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:26:17.153961847 +0000 UTC m=+1811.847937987" watchObservedRunningTime="2026-07-17 20:26:17.155111662 +0000 UTC m=+1811.849087786"
Jul 17 20:26:18.145054 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:18.145010    2572 generic.go:356] "Generic (PLEG): container finished" podID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerID="b7adadec4957aa7baf974191348b7d9e49e4bd2471240e477b739191d92fc1cb" exitCode=6
Jul 17 20:26:18.145444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:18.145082    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" event={"ID":"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd","Type":"ContainerDied","Data":"b7adadec4957aa7baf974191348b7d9e49e4bd2471240e477b739191d92fc1cb"}
Jul 17 20:26:18.145444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:18.145142    2572 scope.go:117] "RemoveContainer" containerID="ad72679bef601e86445a57ee7b53e2c3038a8bb6898da12a967647a7c5bdea2c"
Jul 17 20:26:18.145444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:18.145396    2572 scope.go:117] "RemoveContainer" containerID="b7adadec4957aa7baf974191348b7d9e49e4bd2471240e477b739191d92fc1cb"
Jul 17 20:26:18.145614 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:18.145595    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-jdxxh_odh-ai-gateway-infra(88c4b868-a2a7-4b6c-91c2-105ec7ce57fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd"
Jul 17 20:26:18.166389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:18.166351    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"]
Jul 17 20:26:19.280660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:19.280636    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"
Jul 17 20:26:19.406859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:19.406773    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c4b868-a2a7-4b6c-91c2-105ec7ce57fd-maas-api-service-ca\") pod \"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd\" (UID: \"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd\") "
Jul 17 20:26:19.407163 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:19.407142    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c4b868-a2a7-4b6c-91c2-105ec7ce57fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" (UID: "88c4b868-a2a7-4b6c-91c2-105ec7ce57fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:26:19.508238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:19.508198    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c4b868-a2a7-4b6c-91c2-105ec7ce57fd-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:26:20.155686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:20.155611    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"
Jul 17 20:26:20.155830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:20.155602    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh" event={"ID":"88c4b868-a2a7-4b6c-91c2-105ec7ce57fd","Type":"ContainerDied","Data":"7ed6ce6f6eb966359efc24824aa4574c29b94db5ea892828209a18c08110139e"}
Jul 17 20:26:20.155830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:20.155755    2572 scope.go:117] "RemoveContainer" containerID="b7adadec4957aa7baf974191348b7d9e49e4bd2471240e477b739191d92fc1cb"
Jul 17 20:26:20.171344 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:20.171310    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"]
Jul 17 20:26:20.176945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:20.176923    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-jdxxh"]
Jul 17 20:26:21.853811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:21.853775    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" path="/var/lib/kubelet/pods/88c4b868-a2a7-4b6c-91c2-105ec7ce57fd/volumes"
Jul 17 20:26:26.490648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:26.490609    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:26:28.480159 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480123    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"]
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480429    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480441    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480451    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480457    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480471    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480479    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480524    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:28.480528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480531    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1822bd5-6f65-44a4-8ed9-9ddf9f47a90b" containerName="revoke-keys"
Jul 17 20:26:28.480771 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.480539    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:28.484484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.484466    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"
Jul 17 20:26:28.486834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.486811    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:26:28.486958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.486938    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:26:28.490294 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.490260    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"]
Jul 17 20:26:28.582356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.582317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4c696a-e7b1-4ca1-954d-2d6242588315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd\" (UID: \"bd4c696a-e7b1-4ca1-954d-2d6242588315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"
Jul 17 20:26:28.682753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.682711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4c696a-e7b1-4ca1-954d-2d6242588315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd\" (UID: \"bd4c696a-e7b1-4ca1-954d-2d6242588315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"
Jul 17 20:26:28.683459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.683433    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4c696a-e7b1-4ca1-954d-2d6242588315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd\" (UID: \"bd4c696a-e7b1-4ca1-954d-2d6242588315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"
Jul 17 20:26:28.797252 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.797150    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"
Jul 17 20:26:28.933939 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:28.933904    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"]
Jul 17 20:26:28.936958 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:26:28.936927    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4c696a_e7b1_4ca1_954d_2d6242588315.slice/crio-73570d56ec045a0e284deaddb67faad5bd2939b2f1a62f399553b61b6bae6ab9 WatchSource:0}: Error finding container 73570d56ec045a0e284deaddb67faad5bd2939b2f1a62f399553b61b6bae6ab9: Status 404 returned error can't find the container with id 73570d56ec045a0e284deaddb67faad5bd2939b2f1a62f399553b61b6bae6ab9
Jul 17 20:26:29.196058 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:29.196005    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" event={"ID":"bd4c696a-e7b1-4ca1-954d-2d6242588315","Type":"ContainerStarted","Data":"73570d56ec045a0e284deaddb67faad5bd2939b2f1a62f399553b61b6bae6ab9"}
Jul 17 20:26:30.201705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:30.201670    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerID="e6a0568069b71fec469ae8f8acec9ffad969a3a9850383ca621ddf8cb8ef09f3" exitCode=7
Jul 17 20:26:30.202205 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:30.201730    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" event={"ID":"bd4c696a-e7b1-4ca1-954d-2d6242588315","Type":"ContainerDied","Data":"e6a0568069b71fec469ae8f8acec9ffad969a3a9850383ca621ddf8cb8ef09f3"}
Jul 17 20:26:30.202205 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:30.201964    2572 scope.go:117] "RemoveContainer" containerID="e6a0568069b71fec469ae8f8acec9ffad969a3a9850383ca621ddf8cb8ef09f3"
Jul 17 20:26:31.207925 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:31.207840    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerID="e10827c765c12789442952258d91b4b72e56c01ebada3f7acb4bdf1f8a8565ba" exitCode=7
Jul 17 20:26:31.207925 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:31.207890    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" event={"ID":"bd4c696a-e7b1-4ca1-954d-2d6242588315","Type":"ContainerDied","Data":"e10827c765c12789442952258d91b4b72e56c01ebada3f7acb4bdf1f8a8565ba"}
Jul 17 20:26:31.208496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:31.207948    2572 scope.go:117] "RemoveContainer" containerID="e6a0568069b71fec469ae8f8acec9ffad969a3a9850383ca621ddf8cb8ef09f3"
Jul 17 20:26:31.208496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:31.208178    2572 scope.go:117] "RemoveContainer" containerID="e10827c765c12789442952258d91b4b72e56c01ebada3f7acb4bdf1f8a8565ba"
Jul 17 20:26:31.208496 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:31.208371    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd_odh-ai-gateway-infra(bd4c696a-e7b1-4ca1-954d-2d6242588315)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315"
Jul 17 20:26:32.213573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:32.213547    2572 scope.go:117] "RemoveContainer" containerID="e10827c765c12789442952258d91b4b72e56c01ebada3f7acb4bdf1f8a8565ba"
Jul 17 20:26:32.213979 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:32.213705    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd_odh-ai-gateway-infra(bd4c696a-e7b1-4ca1-954d-2d6242588315)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315"
Jul 17 20:26:42.848647 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:42.848615    2572 scope.go:117] "RemoveContainer" containerID="e10827c765c12789442952258d91b4b72e56c01ebada3f7acb4bdf1f8a8565ba"
Jul 17 20:26:43.262160 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:43.262123    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" event={"ID":"bd4c696a-e7b1-4ca1-954d-2d6242588315","Type":"ContainerStarted","Data":"55c00352fc5a798f3194e78f4c014245cf77753559ae4d59b738fe36b8bb59fe"}
Jul 17 20:26:43.278379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:43.278323    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" podStartSLOduration=15.278306847 podStartE2EDuration="15.278306847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:26:43.278276733 +0000 UTC m=+1837.972252880" watchObservedRunningTime="2026-07-17 20:26:43.278306847 +0000 UTC m=+1837.972282992"
Jul 17 20:26:44.268537 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:44.268501    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerID="55c00352fc5a798f3194e78f4c014245cf77753559ae4d59b738fe36b8bb59fe" exitCode=7
Jul 17 20:26:44.268967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:44.268572    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" event={"ID":"bd4c696a-e7b1-4ca1-954d-2d6242588315","Type":"ContainerDied","Data":"55c00352fc5a798f3194e78f4c014245cf77753559ae4d59b738fe36b8bb59fe"}
Jul 17 20:26:44.268967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:44.268631    2572 scope.go:117] "RemoveContainer" containerID="e10827c765c12789442952258d91b4b72e56c01ebada3f7acb4bdf1f8a8565ba"
Jul 17 20:26:44.268967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:44.268900    2572 scope.go:117] "RemoveContainer" containerID="55c00352fc5a798f3194e78f4c014245cf77753559ae4d59b738fe36b8bb59fe"
Jul 17 20:26:44.269127 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:44.269108    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd_odh-ai-gateway-infra(bd4c696a-e7b1-4ca1-954d-2d6242588315)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315"
Jul 17 20:26:44.288976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:44.288945    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"]
Jul 17 20:26:45.400854 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:45.400831    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"
Jul 17 20:26:45.413393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:45.413371    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4c696a-e7b1-4ca1-954d-2d6242588315-maas-api-service-ca\") pod \"bd4c696a-e7b1-4ca1-954d-2d6242588315\" (UID: \"bd4c696a-e7b1-4ca1-954d-2d6242588315\") "
Jul 17 20:26:45.413701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:45.413680    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4c696a-e7b1-4ca1-954d-2d6242588315-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd4c696a-e7b1-4ca1-954d-2d6242588315" (UID: "bd4c696a-e7b1-4ca1-954d-2d6242588315"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:26:45.514287 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:45.514246    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4c696a-e7b1-4ca1-954d-2d6242588315-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:26:46.279442 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:46.279415    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd" event={"ID":"bd4c696a-e7b1-4ca1-954d-2d6242588315","Type":"ContainerDied","Data":"73570d56ec045a0e284deaddb67faad5bd2939b2f1a62f399553b61b6bae6ab9"}
Jul 17 20:26:46.279442 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:46.279440    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"
Jul 17 20:26:46.279672 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:46.279468    2572 scope.go:117] "RemoveContainer" containerID="55c00352fc5a798f3194e78f4c014245cf77753559ae4d59b738fe36b8bb59fe"
Jul 17 20:26:46.295030 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:46.295006    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"]
Jul 17 20:26:46.296495 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:46.296478    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nv7gd"]
Jul 17 20:26:47.853112 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:47.853084    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" path="/var/lib/kubelet/pods/bd4c696a-e7b1-4ca1-954d-2d6242588315/volumes"
Jul 17 20:26:50.507032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.506990    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"]
Jul 17 20:26:50.507502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507427    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:50.507502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507445    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:50.507502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507463    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:26:50.507502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507471    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:26:50.507502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507494    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:26:50.507502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507502    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:26:50.507799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507575    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c4b868-a2a7-4b6c-91c2-105ec7ce57fd" containerName="revoke-keys"
Jul 17 20:26:50.507799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507586    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:26:50.507799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507596    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:26:50.507799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.507606    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:26:50.511908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.511887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"
Jul 17 20:26:50.514287 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.514267    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:26:50.514393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.514269    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:26:50.517542 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.517505    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"]
Jul 17 20:26:50.556783 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.556754    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-v7tc4\" (UID: \"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"
Jul 17 20:26:50.657608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.657570    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-v7tc4\" (UID: \"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"
Jul 17 20:26:50.658240 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.658216    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-v7tc4\" (UID: \"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"
Jul 17 20:26:50.823889 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.823801    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"
Jul 17 20:26:50.955152 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:50.955123    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"]
Jul 17 20:26:50.958220 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:26:50.958195    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aac6cb7_3d65_4ae0_b807_73dabc8f1e4e.slice/crio-ea88f5db792279f9f1b7e52ae69bb742915bf28ae03e36dd3c207bac82d3e413 WatchSource:0}: Error finding container ea88f5db792279f9f1b7e52ae69bb742915bf28ae03e36dd3c207bac82d3e413: Status 404 returned error can't find the container with id ea88f5db792279f9f1b7e52ae69bb742915bf28ae03e36dd3c207bac82d3e413
Jul 17 20:26:51.302845 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:51.302810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" event={"ID":"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e","Type":"ContainerStarted","Data":"d20f19d8eda7078de23be247cf8bd673a69019113d41960fa057d97027811f2e"}
Jul 17 20:26:51.303024 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:51.302869    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" event={"ID":"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e","Type":"ContainerStarted","Data":"ea88f5db792279f9f1b7e52ae69bb742915bf28ae03e36dd3c207bac82d3e413"}
Jul 17 20:26:51.317320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:51.317267    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" podStartSLOduration=1.317250576 podStartE2EDuration="1.317250576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:26:51.316408573 +0000 UTC m=+1846.010384721" watchObservedRunningTime="2026-07-17 20:26:51.317250576 +0000 UTC m=+1846.011226720"
Jul 17 20:26:52.308391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:52.308355    2572 generic.go:356] "Generic (PLEG): container finished" podID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerID="d20f19d8eda7078de23be247cf8bd673a69019113d41960fa057d97027811f2e" exitCode=6
Jul 17 20:26:52.308767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:52.308405    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" event={"ID":"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e","Type":"ContainerDied","Data":"d20f19d8eda7078de23be247cf8bd673a69019113d41960fa057d97027811f2e"}
Jul 17 20:26:52.308767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:52.308604    2572 scope.go:117] "RemoveContainer" containerID="d20f19d8eda7078de23be247cf8bd673a69019113d41960fa057d97027811f2e"
Jul 17 20:26:53.313908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:53.313873    2572 generic.go:356] "Generic (PLEG): container finished" podID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerID="2501a8936f6371811f6a42c1d4b16911263f9651134e4b4a42f7f005f135bdfc" exitCode=6
Jul 17 20:26:53.314360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:53.313960    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" event={"ID":"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e","Type":"ContainerDied","Data":"2501a8936f6371811f6a42c1d4b16911263f9651134e4b4a42f7f005f135bdfc"}
Jul 17 20:26:53.314360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:53.314014    2572 scope.go:117] "RemoveContainer" containerID="d20f19d8eda7078de23be247cf8bd673a69019113d41960fa057d97027811f2e"
Jul 17 20:26:53.314360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:53.314193    2572 scope.go:117] "RemoveContainer" containerID="2501a8936f6371811f6a42c1d4b16911263f9651134e4b4a42f7f005f135bdfc"
Jul 17 20:26:53.314467 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:53.314414    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-v7tc4_odh-ai-gateway-infra(8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e"
Jul 17 20:26:54.319286 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:26:54.319262    2572 scope.go:117] "RemoveContainer" containerID="2501a8936f6371811f6a42c1d4b16911263f9651134e4b4a42f7f005f135bdfc"
Jul 17 20:26:54.319741 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:26:54.319485    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-v7tc4_odh-ai-gateway-infra(8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e"
Jul 17 20:27:04.847740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:04.847712    2572 scope.go:117] "RemoveContainer" containerID="2501a8936f6371811f6a42c1d4b16911263f9651134e4b4a42f7f005f135bdfc"
Jul 17 20:27:05.367745 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:05.367710    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" event={"ID":"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e","Type":"ContainerStarted","Data":"2c9bea6b0535a8c928bd264bf6e68572277e69452969eb2bdedd82e73ac6f705"}
Jul 17 20:27:06.373427 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:06.373393    2572 generic.go:356] "Generic (PLEG): container finished" podID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerID="2c9bea6b0535a8c928bd264bf6e68572277e69452969eb2bdedd82e73ac6f705" exitCode=6
Jul 17 20:27:06.373833 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:06.373466    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" event={"ID":"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e","Type":"ContainerDied","Data":"2c9bea6b0535a8c928bd264bf6e68572277e69452969eb2bdedd82e73ac6f705"}
Jul 17 20:27:06.373833 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:06.373525    2572 scope.go:117] "RemoveContainer" containerID="2501a8936f6371811f6a42c1d4b16911263f9651134e4b4a42f7f005f135bdfc"
Jul 17 20:27:06.373833 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:06.373769    2572 scope.go:117] "RemoveContainer" containerID="2c9bea6b0535a8c928bd264bf6e68572277e69452969eb2bdedd82e73ac6f705"
Jul 17 20:27:06.373992 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:06.373974    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-v7tc4_odh-ai-gateway-infra(8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e"
Jul 17 20:27:06.395535 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:06.395506    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"]
Jul 17 20:27:07.510151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:07.510129    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"
Jul 17 20:27:07.581328 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:07.581285    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e-maas-api-service-ca\") pod \"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e\" (UID: \"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e\") "
Jul 17 20:27:07.581673 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:07.581651    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" (UID: "8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:27:07.682239 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:07.682163    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:27:08.385470 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:08.385441    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"
Jul 17 20:27:08.385696 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:08.385466    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4" event={"ID":"8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e","Type":"ContainerDied","Data":"ea88f5db792279f9f1b7e52ae69bb742915bf28ae03e36dd3c207bac82d3e413"}
Jul 17 20:27:08.385696 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:08.385526    2572 scope.go:117] "RemoveContainer" containerID="2c9bea6b0535a8c928bd264bf6e68572277e69452969eb2bdedd82e73ac6f705"
Jul 17 20:27:08.403581 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:08.403558    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"]
Jul 17 20:27:08.405613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:08.405592    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-v7tc4"]
Jul 17 20:27:09.853627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:09.853598    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" path="/var/lib/kubelet/pods/8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e/volumes"
Jul 17 20:27:18.529247 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529215    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"]
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529505    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529515    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529531    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529536    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4c696a-e7b1-4ca1-954d-2d6242588315" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529545    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529551    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529600    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529607    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:18.529616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.529613    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:18.533547 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.533528    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"
Jul 17 20:27:18.536260 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.536241    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:27:18.536313 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.536244    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:27:18.540657 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.540632    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"]
Jul 17 20:27:18.672426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.672395    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac165276-75f8-4b61-ada8-da50a4d154cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5\" (UID: \"ac165276-75f8-4b61-ada8-da50a4d154cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"
Jul 17 20:27:18.773103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.773069    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac165276-75f8-4b61-ada8-da50a4d154cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5\" (UID: \"ac165276-75f8-4b61-ada8-da50a4d154cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"
Jul 17 20:27:18.773612 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.773591    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac165276-75f8-4b61-ada8-da50a4d154cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5\" (UID: \"ac165276-75f8-4b61-ada8-da50a4d154cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"
Jul 17 20:27:18.844817 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.844735    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"
Jul 17 20:27:18.971318 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:18.971296    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"]
Jul 17 20:27:18.973263 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:27:18.973235    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac165276_75f8_4b61_ada8_da50a4d154cd.slice/crio-fdd481d3c53e9ef9b7b28257a5dbde1cab423a79e48618217d65df1a98e7c594 WatchSource:0}: Error finding container fdd481d3c53e9ef9b7b28257a5dbde1cab423a79e48618217d65df1a98e7c594: Status 404 returned error can't find the container with id fdd481d3c53e9ef9b7b28257a5dbde1cab423a79e48618217d65df1a98e7c594
Jul 17 20:27:19.432573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:19.432488    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" event={"ID":"ac165276-75f8-4b61-ada8-da50a4d154cd","Type":"ContainerStarted","Data":"da3d709feb30b005fbeb6027dc015b79a846dff43b7baf85f7945aae75afa522"}
Jul 17 20:27:19.432573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:19.432542    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" event={"ID":"ac165276-75f8-4b61-ada8-da50a4d154cd","Type":"ContainerStarted","Data":"fdd481d3c53e9ef9b7b28257a5dbde1cab423a79e48618217d65df1a98e7c594"}
Jul 17 20:27:19.452205 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:19.452158    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" podStartSLOduration=1.452143553 podStartE2EDuration="1.452143553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:27:19.451417245 +0000 UTC m=+1874.145393390" watchObservedRunningTime="2026-07-17 20:27:19.452143553 +0000 UTC m=+1874.146119699"
Jul 17 20:27:20.438010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:20.437975    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerID="da3d709feb30b005fbeb6027dc015b79a846dff43b7baf85f7945aae75afa522" exitCode=7
Jul 17 20:27:20.438477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:20.438033    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" event={"ID":"ac165276-75f8-4b61-ada8-da50a4d154cd","Type":"ContainerDied","Data":"da3d709feb30b005fbeb6027dc015b79a846dff43b7baf85f7945aae75afa522"}
Jul 17 20:27:20.438477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:20.438290    2572 scope.go:117] "RemoveContainer" containerID="da3d709feb30b005fbeb6027dc015b79a846dff43b7baf85f7945aae75afa522"
Jul 17 20:27:21.444324 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:21.444238    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerID="5af575ecc0b150f9a1335e665385c57e4bcf17a2c622945671bc74a5e689fe9b" exitCode=7
Jul 17 20:27:21.444765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:21.444327    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" event={"ID":"ac165276-75f8-4b61-ada8-da50a4d154cd","Type":"ContainerDied","Data":"5af575ecc0b150f9a1335e665385c57e4bcf17a2c622945671bc74a5e689fe9b"}
Jul 17 20:27:21.444765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:21.444388    2572 scope.go:117] "RemoveContainer" containerID="da3d709feb30b005fbeb6027dc015b79a846dff43b7baf85f7945aae75afa522"
Jul 17 20:27:21.444765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:21.444597    2572 scope.go:117] "RemoveContainer" containerID="5af575ecc0b150f9a1335e665385c57e4bcf17a2c622945671bc74a5e689fe9b"
Jul 17 20:27:21.444915 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:21.444818    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5_odh-ai-gateway-infra(ac165276-75f8-4b61-ada8-da50a4d154cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd"
Jul 17 20:27:22.450466 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:22.450439    2572 scope.go:117] "RemoveContainer" containerID="5af575ecc0b150f9a1335e665385c57e4bcf17a2c622945671bc74a5e689fe9b"
Jul 17 20:27:22.450838 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:22.450613    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5_odh-ai-gateway-infra(ac165276-75f8-4b61-ada8-da50a4d154cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd"
Jul 17 20:27:31.280648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:31.280566    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:27:33.848629 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:33.848598    2572 scope.go:117] "RemoveContainer" containerID="5af575ecc0b150f9a1335e665385c57e4bcf17a2c622945671bc74a5e689fe9b"
Jul 17 20:27:34.500612 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:34.500578    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" event={"ID":"ac165276-75f8-4b61-ada8-da50a4d154cd","Type":"ContainerStarted","Data":"b57129fcbc6d0c8c5bb93ccf5393e68685ebffde63611ade1ea0e786898fe97e"}
Jul 17 20:27:35.506984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:35.506949    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerID="b57129fcbc6d0c8c5bb93ccf5393e68685ebffde63611ade1ea0e786898fe97e" exitCode=6
Jul 17 20:27:35.507402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:35.507023    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" event={"ID":"ac165276-75f8-4b61-ada8-da50a4d154cd","Type":"ContainerDied","Data":"b57129fcbc6d0c8c5bb93ccf5393e68685ebffde63611ade1ea0e786898fe97e"}
Jul 17 20:27:35.507402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:35.507096    2572 scope.go:117] "RemoveContainer" containerID="5af575ecc0b150f9a1335e665385c57e4bcf17a2c622945671bc74a5e689fe9b"
Jul 17 20:27:35.507402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:35.507355    2572 scope.go:117] "RemoveContainer" containerID="b57129fcbc6d0c8c5bb93ccf5393e68685ebffde63611ade1ea0e786898fe97e"
Jul 17 20:27:35.507569 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:35.507550    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5_odh-ai-gateway-infra(ac165276-75f8-4b61-ada8-da50a4d154cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd"
Jul 17 20:27:35.536336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:35.536306    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"]
Jul 17 20:27:36.644884 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:36.644856    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"
Jul 17 20:27:36.824324 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:36.824234    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac165276-75f8-4b61-ada8-da50a4d154cd-maas-api-service-ca\") pod \"ac165276-75f8-4b61-ada8-da50a4d154cd\" (UID: \"ac165276-75f8-4b61-ada8-da50a4d154cd\") "
Jul 17 20:27:36.824595 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:36.824570    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac165276-75f8-4b61-ada8-da50a4d154cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac165276-75f8-4b61-ada8-da50a4d154cd" (UID: "ac165276-75f8-4b61-ada8-da50a4d154cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:27:36.925021 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:36.924983    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac165276-75f8-4b61-ada8-da50a4d154cd-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:27:37.517511 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:37.517487    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"
Jul 17 20:27:37.517673 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:37.517484    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5" event={"ID":"ac165276-75f8-4b61-ada8-da50a4d154cd","Type":"ContainerDied","Data":"fdd481d3c53e9ef9b7b28257a5dbde1cab423a79e48618217d65df1a98e7c594"}
Jul 17 20:27:37.517673 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:37.517618    2572 scope.go:117] "RemoveContainer" containerID="b57129fcbc6d0c8c5bb93ccf5393e68685ebffde63611ade1ea0e786898fe97e"
Jul 17 20:27:37.538004 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:37.537975    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"]
Jul 17 20:27:37.542638 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:37.542617    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-m6qg5"]
Jul 17 20:27:37.853013 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:37.852938    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" path="/var/lib/kubelet/pods/ac165276-75f8-4b61-ada8-da50a4d154cd/volumes"
Jul 17 20:27:40.557842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.557807    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"]
Jul 17 20:27:40.558267 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558253    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:40.558310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558271    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aac6cb7-3d65-4ae0-b807-73dabc8f1e4e" containerName="revoke-keys"
Jul 17 20:27:40.558310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558282    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558290    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558308    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558317    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558333    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558341    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558414    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558425    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.558440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.558434    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac165276-75f8-4b61-ada8-da50a4d154cd" containerName="revoke-keys"
Jul 17 20:27:40.561523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.561503    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"
Jul 17 20:27:40.563866 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.563844    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:27:40.564035 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.563920    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:27:40.570286 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.569478    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"]
Jul 17 20:27:40.655829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.655786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af26dcc6-5596-4174-96d4-de1d635008ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-vdvxm\" (UID: \"af26dcc6-5596-4174-96d4-de1d635008ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"
Jul 17 20:27:40.756762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.756715    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af26dcc6-5596-4174-96d4-de1d635008ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-vdvxm\" (UID: \"af26dcc6-5596-4174-96d4-de1d635008ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"
Jul 17 20:27:40.757382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.757362    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af26dcc6-5596-4174-96d4-de1d635008ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-vdvxm\" (UID: \"af26dcc6-5596-4174-96d4-de1d635008ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"
Jul 17 20:27:40.880741 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:40.880668    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"
Jul 17 20:27:41.011528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:41.011502    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"]
Jul 17 20:27:41.013559 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:27:41.013530    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf26dcc6_5596_4174_96d4_de1d635008ee.slice/crio-b29ceb031382f86ee489f9fe5fcbd2d83f7a10d4d471a6eab7731ba419a8938c WatchSource:0}: Error finding container b29ceb031382f86ee489f9fe5fcbd2d83f7a10d4d471a6eab7731ba419a8938c: Status 404 returned error can't find the container with id b29ceb031382f86ee489f9fe5fcbd2d83f7a10d4d471a6eab7731ba419a8938c
Jul 17 20:27:41.536423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:41.536337    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" event={"ID":"af26dcc6-5596-4174-96d4-de1d635008ee","Type":"ContainerStarted","Data":"915de628f9c33780d48feb3812be82dfce525f4ac5ad7a26f9f425e80302f72d"}
Jul 17 20:27:41.536423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:41.536391    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" event={"ID":"af26dcc6-5596-4174-96d4-de1d635008ee","Type":"ContainerStarted","Data":"b29ceb031382f86ee489f9fe5fcbd2d83f7a10d4d471a6eab7731ba419a8938c"}
Jul 17 20:27:41.567441 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:41.567388    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" podStartSLOduration=1.5673711350000001 podStartE2EDuration="1.567371135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:27:41.565637544 +0000 UTC m=+1896.259613692" watchObservedRunningTime="2026-07-17 20:27:41.567371135 +0000 UTC m=+1896.261347280"
Jul 17 20:27:42.542080 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:42.542029    2572 generic.go:356] "Generic (PLEG): container finished" podID="af26dcc6-5596-4174-96d4-de1d635008ee" containerID="915de628f9c33780d48feb3812be82dfce525f4ac5ad7a26f9f425e80302f72d" exitCode=6
Jul 17 20:27:42.542249 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:42.542095    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" event={"ID":"af26dcc6-5596-4174-96d4-de1d635008ee","Type":"ContainerDied","Data":"915de628f9c33780d48feb3812be82dfce525f4ac5ad7a26f9f425e80302f72d"}
Jul 17 20:27:42.542303 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:42.542269    2572 scope.go:117] "RemoveContainer" containerID="915de628f9c33780d48feb3812be82dfce525f4ac5ad7a26f9f425e80302f72d"
Jul 17 20:27:43.549731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:43.549651    2572 generic.go:356] "Generic (PLEG): container finished" podID="af26dcc6-5596-4174-96d4-de1d635008ee" containerID="e40fa9305ce155d3fb64137fcdf0a23e0dd718321cbc8818ebe68de57167ae57" exitCode=6
Jul 17 20:27:43.549731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:43.549704    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" event={"ID":"af26dcc6-5596-4174-96d4-de1d635008ee","Type":"ContainerDied","Data":"e40fa9305ce155d3fb64137fcdf0a23e0dd718321cbc8818ebe68de57167ae57"}
Jul 17 20:27:43.550199 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:43.549778    2572 scope.go:117] "RemoveContainer" containerID="915de628f9c33780d48feb3812be82dfce525f4ac5ad7a26f9f425e80302f72d"
Jul 17 20:27:43.550199 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:43.549946    2572 scope.go:117] "RemoveContainer" containerID="e40fa9305ce155d3fb64137fcdf0a23e0dd718321cbc8818ebe68de57167ae57"
Jul 17 20:27:43.550199 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:43.550184    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-vdvxm_odh-ai-gateway-infra(af26dcc6-5596-4174-96d4-de1d635008ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" podUID="af26dcc6-5596-4174-96d4-de1d635008ee"
Jul 17 20:27:44.479720 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:44.479682    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:27:44.558447 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:44.558421    2572 scope.go:117] "RemoveContainer" containerID="e40fa9305ce155d3fb64137fcdf0a23e0dd718321cbc8818ebe68de57167ae57"
Jul 17 20:27:44.558908 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:44.558606    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-vdvxm_odh-ai-gateway-infra(af26dcc6-5596-4174-96d4-de1d635008ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" podUID="af26dcc6-5596-4174-96d4-de1d635008ee"
Jul 17 20:27:46.785214 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:46.785180    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"]
Jul 17 20:27:46.789507 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:46.789490    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"
Jul 17 20:27:46.796783 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:46.796427    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"]
Jul 17 20:27:46.809368 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:46.809339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9945c2-6d07-4618-b62e-ec2d1a1b9da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5\" (UID: \"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"
Jul 17 20:27:46.910226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:46.910189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9945c2-6d07-4618-b62e-ec2d1a1b9da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5\" (UID: \"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"
Jul 17 20:27:46.910796 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:46.910776    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9945c2-6d07-4618-b62e-ec2d1a1b9da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5\" (UID: \"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"
Jul 17 20:27:47.100956 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:47.100876    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"
Jul 17 20:27:47.228984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:47.228955    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"]
Jul 17 20:27:47.231949 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:27:47.231918    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9945c2_6d07_4618_b62e_ec2d1a1b9da5.slice/crio-7e7eb5f3007128fd1e0d3000539ebcf9df9e6d4ea5b3174b9749fbe8d9fbf247 WatchSource:0}: Error finding container 7e7eb5f3007128fd1e0d3000539ebcf9df9e6d4ea5b3174b9749fbe8d9fbf247: Status 404 returned error can't find the container with id 7e7eb5f3007128fd1e0d3000539ebcf9df9e6d4ea5b3174b9749fbe8d9fbf247
Jul 17 20:27:47.572712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:47.572678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" event={"ID":"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5","Type":"ContainerStarted","Data":"e52d7d3575224ddc983f9d79e09bc6f3e0dfb20b1a085916f364a9122bec291b"}
Jul 17 20:27:47.572894 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:47.572728    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" event={"ID":"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5","Type":"ContainerStarted","Data":"7e7eb5f3007128fd1e0d3000539ebcf9df9e6d4ea5b3174b9749fbe8d9fbf247"}
Jul 17 20:27:47.605879 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:47.605826    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" podStartSLOduration=1.6058100990000002 podStartE2EDuration="1.605810099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:27:47.605205594 +0000 UTC m=+1902.299181740" watchObservedRunningTime="2026-07-17 20:27:47.605810099 +0000 UTC m=+1902.299786245"
Jul 17 20:27:48.578565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:48.578524    2572 generic.go:356] "Generic (PLEG): container finished" podID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerID="e52d7d3575224ddc983f9d79e09bc6f3e0dfb20b1a085916f364a9122bec291b" exitCode=7
Jul 17 20:27:48.578934 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:48.578618    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" event={"ID":"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5","Type":"ContainerDied","Data":"e52d7d3575224ddc983f9d79e09bc6f3e0dfb20b1a085916f364a9122bec291b"}
Jul 17 20:27:48.578934 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:48.578813    2572 scope.go:117] "RemoveContainer" containerID="e52d7d3575224ddc983f9d79e09bc6f3e0dfb20b1a085916f364a9122bec291b"
Jul 17 20:27:49.585004 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:49.584973    2572 generic.go:356] "Generic (PLEG): container finished" podID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerID="02c93d6c3ab142825fb2f6c3d0f50f46274155840c4770d2aac680c6b420bf41" exitCode=7
Jul 17 20:27:49.585426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:49.585031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" event={"ID":"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5","Type":"ContainerDied","Data":"02c93d6c3ab142825fb2f6c3d0f50f46274155840c4770d2aac680c6b420bf41"}
Jul 17 20:27:49.585426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:49.585091    2572 scope.go:117] "RemoveContainer" containerID="e52d7d3575224ddc983f9d79e09bc6f3e0dfb20b1a085916f364a9122bec291b"
Jul 17 20:27:49.585426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:49.585248    2572 scope.go:117] "RemoveContainer" containerID="02c93d6c3ab142825fb2f6c3d0f50f46274155840c4770d2aac680c6b420bf41"
Jul 17 20:27:49.585531 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:49.585469    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5_odh-ai-gateway-infra(2e9945c2-6d07-4618-b62e-ec2d1a1b9da5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5"
Jul 17 20:27:50.590963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:50.590936    2572 scope.go:117] "RemoveContainer" containerID="02c93d6c3ab142825fb2f6c3d0f50f46274155840c4770d2aac680c6b420bf41"
Jul 17 20:27:50.591392 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:27:50.591136    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5_odh-ai-gateway-infra(2e9945c2-6d07-4618-b62e-ec2d1a1b9da5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5"
Jul 17 20:27:59.848397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:27:59.848364    2572 scope.go:117] "RemoveContainer" containerID="e40fa9305ce155d3fb64137fcdf0a23e0dd718321cbc8818ebe68de57167ae57"
Jul 17 20:28:00.634571 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:00.634528    2572 generic.go:356] "Generic (PLEG): container finished" podID="af26dcc6-5596-4174-96d4-de1d635008ee" containerID="59ed8f0119fe7ea1ae162ae16ba2df19b467b3c1b13e6b50b9acbe762836a96c" exitCode=6
Jul 17 20:28:00.634729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:00.634603    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" event={"ID":"af26dcc6-5596-4174-96d4-de1d635008ee","Type":"ContainerDied","Data":"59ed8f0119fe7ea1ae162ae16ba2df19b467b3c1b13e6b50b9acbe762836a96c"}
Jul 17 20:28:00.634729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:00.634661    2572 scope.go:117] "RemoveContainer" containerID="e40fa9305ce155d3fb64137fcdf0a23e0dd718321cbc8818ebe68de57167ae57"
Jul 17 20:28:00.634938 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:00.634923    2572 scope.go:117] "RemoveContainer" containerID="59ed8f0119fe7ea1ae162ae16ba2df19b467b3c1b13e6b50b9acbe762836a96c"
Jul 17 20:28:00.635138 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:00.635119    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-vdvxm_odh-ai-gateway-infra(af26dcc6-5596-4174-96d4-de1d635008ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" podUID="af26dcc6-5596-4174-96d4-de1d635008ee"
Jul 17 20:28:01.659529 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:01.659491    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"]
Jul 17 20:28:01.791945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:01.791924    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"
Jul 17 20:28:01.826292 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:01.826259    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af26dcc6-5596-4174-96d4-de1d635008ee-maas-api-service-ca\") pod \"af26dcc6-5596-4174-96d4-de1d635008ee\" (UID: \"af26dcc6-5596-4174-96d4-de1d635008ee\") "
Jul 17 20:28:01.826688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:01.826667    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af26dcc6-5596-4174-96d4-de1d635008ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af26dcc6-5596-4174-96d4-de1d635008ee" (UID: "af26dcc6-5596-4174-96d4-de1d635008ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:28:01.927382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:01.927278    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af26dcc6-5596-4174-96d4-de1d635008ee-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:28:02.645967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:02.645935    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"
Jul 17 20:28:02.646237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:02.645934    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm" event={"ID":"af26dcc6-5596-4174-96d4-de1d635008ee","Type":"ContainerDied","Data":"b29ceb031382f86ee489f9fe5fcbd2d83f7a10d4d471a6eab7731ba419a8938c"}
Jul 17 20:28:02.646237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:02.646090    2572 scope.go:117] "RemoveContainer" containerID="59ed8f0119fe7ea1ae162ae16ba2df19b467b3c1b13e6b50b9acbe762836a96c"
Jul 17 20:28:02.661829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:02.661806    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"]
Jul 17 20:28:02.665684 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:02.665658    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-vdvxm"]
Jul 17 20:28:02.848502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:02.848470    2572 scope.go:117] "RemoveContainer" containerID="02c93d6c3ab142825fb2f6c3d0f50f46274155840c4770d2aac680c6b420bf41"
Jul 17 20:28:03.651385 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:03.651357    2572 generic.go:356] "Generic (PLEG): container finished" podID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerID="98ee498d7f901db2e5a6b09a162c2e0cb887f15d8de2fe3c1cebb61fd12db302" exitCode=7
Jul 17 20:28:03.651565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:03.651431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" event={"ID":"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5","Type":"ContainerDied","Data":"98ee498d7f901db2e5a6b09a162c2e0cb887f15d8de2fe3c1cebb61fd12db302"}
Jul 17 20:28:03.651565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:03.651493    2572 scope.go:117] "RemoveContainer" containerID="02c93d6c3ab142825fb2f6c3d0f50f46274155840c4770d2aac680c6b420bf41"
Jul 17 20:28:03.651810 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:03.651791    2572 scope.go:117] "RemoveContainer" containerID="98ee498d7f901db2e5a6b09a162c2e0cb887f15d8de2fe3c1cebb61fd12db302"
Jul 17 20:28:03.652002 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:03.651982    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5_odh-ai-gateway-infra(2e9945c2-6d07-4618-b62e-ec2d1a1b9da5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5"
Jul 17 20:28:03.853420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:03.853383    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" path="/var/lib/kubelet/pods/af26dcc6-5596-4174-96d4-de1d635008ee/volumes"
Jul 17 20:28:04.675916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:04.675880    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"]
Jul 17 20:28:04.809993 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:04.809972    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"
Jul 17 20:28:04.848879 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:04.848853    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9945c2-6d07-4618-b62e-ec2d1a1b9da5-maas-api-service-ca\") pod \"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5\" (UID: \"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5\") "
Jul 17 20:28:04.849198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:04.849178    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9945c2-6d07-4618-b62e-ec2d1a1b9da5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" (UID: "2e9945c2-6d07-4618-b62e-ec2d1a1b9da5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:28:04.949382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:04.949297    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9945c2-6d07-4618-b62e-ec2d1a1b9da5-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:28:05.662669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:05.662640    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"
Jul 17 20:28:05.662897 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:05.662640    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5" event={"ID":"2e9945c2-6d07-4618-b62e-ec2d1a1b9da5","Type":"ContainerDied","Data":"7e7eb5f3007128fd1e0d3000539ebcf9df9e6d4ea5b3174b9749fbe8d9fbf247"}
Jul 17 20:28:05.662897 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:05.662780    2572 scope.go:117] "RemoveContainer" containerID="98ee498d7f901db2e5a6b09a162c2e0cb887f15d8de2fe3c1cebb61fd12db302"
Jul 17 20:28:05.688090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:05.688041    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"]
Jul 17 20:28:05.693592 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:05.693568    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-dhvx5"]
Jul 17 20:28:05.853956 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:05.853926    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" path="/var/lib/kubelet/pods/2e9945c2-6d07-4618-b62e-ec2d1a1b9da5/volumes"
Jul 17 20:28:08.576613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576584    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"]
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576884    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576894    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576907    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576912    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576922    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576927    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576935    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576941    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576947    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.576953    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.577000    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:08.577005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.577009    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.577015    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:08.577379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.577022    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:08.579869 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.579854    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"
Jul 17 20:28:08.583132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.583112    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:28:08.583221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.583112    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:28:08.588405 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.588385    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"]
Jul 17 20:28:08.683099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.683040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68f2f07-c17d-4579-912b-e19ca3f1dd05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz\" (UID: \"a68f2f07-c17d-4579-912b-e19ca3f1dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"
Jul 17 20:28:08.784220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.784180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68f2f07-c17d-4579-912b-e19ca3f1dd05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz\" (UID: \"a68f2f07-c17d-4579-912b-e19ca3f1dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"
Jul 17 20:28:08.784802 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.784778    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68f2f07-c17d-4579-912b-e19ca3f1dd05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz\" (UID: \"a68f2f07-c17d-4579-912b-e19ca3f1dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"
Jul 17 20:28:08.889943 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:08.889869    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"
Jul 17 20:28:09.020293 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:09.020260    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"]
Jul 17 20:28:09.023025 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:28:09.022994    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68f2f07_c17d_4579_912b_e19ca3f1dd05.slice/crio-e7015e87a406c7c5244aa82c9a6489213932a2bc60543aa83a62d5175f81c524 WatchSource:0}: Error finding container e7015e87a406c7c5244aa82c9a6489213932a2bc60543aa83a62d5175f81c524: Status 404 returned error can't find the container with id e7015e87a406c7c5244aa82c9a6489213932a2bc60543aa83a62d5175f81c524
Jul 17 20:28:09.680189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:09.680159    2572 generic.go:356] "Generic (PLEG): container finished" podID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerID="5af746e4092a96b3efe81cebfa04667bfcb46e576e98b20451c62e356b5cbded" exitCode=6
Jul 17 20:28:09.680572 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:09.680236    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" event={"ID":"a68f2f07-c17d-4579-912b-e19ca3f1dd05","Type":"ContainerDied","Data":"5af746e4092a96b3efe81cebfa04667bfcb46e576e98b20451c62e356b5cbded"}
Jul 17 20:28:09.680572 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:09.680283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" event={"ID":"a68f2f07-c17d-4579-912b-e19ca3f1dd05","Type":"ContainerStarted","Data":"e7015e87a406c7c5244aa82c9a6489213932a2bc60543aa83a62d5175f81c524"}
Jul 17 20:28:09.680572 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:09.680378    2572 scope.go:117] "RemoveContainer" containerID="5af746e4092a96b3efe81cebfa04667bfcb46e576e98b20451c62e356b5cbded"
Jul 17 20:28:10.686069 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:10.685956    2572 generic.go:356] "Generic (PLEG): container finished" podID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerID="3751e631c697fe4f8513e174fcd261d4d6eacb6fcb5d0013498d566a0135af47" exitCode=6
Jul 17 20:28:10.686069 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:10.686035    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" event={"ID":"a68f2f07-c17d-4579-912b-e19ca3f1dd05","Type":"ContainerDied","Data":"3751e631c697fe4f8513e174fcd261d4d6eacb6fcb5d0013498d566a0135af47"}
Jul 17 20:28:10.686486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:10.686110    2572 scope.go:117] "RemoveContainer" containerID="5af746e4092a96b3efe81cebfa04667bfcb46e576e98b20451c62e356b5cbded"
Jul 17 20:28:10.686486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:10.686295    2572 scope.go:117] "RemoveContainer" containerID="3751e631c697fe4f8513e174fcd261d4d6eacb6fcb5d0013498d566a0135af47"
Jul 17 20:28:10.686578 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:10.686515    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz_odh-ai-gateway-infra(a68f2f07-c17d-4579-912b-e19ca3f1dd05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05"
Jul 17 20:28:11.691918 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:11.691890    2572 scope.go:117] "RemoveContainer" containerID="3751e631c697fe4f8513e174fcd261d4d6eacb6fcb5d0013498d566a0135af47"
Jul 17 20:28:11.692315 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:11.692096    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz_odh-ai-gateway-infra(a68f2f07-c17d-4579-912b-e19ca3f1dd05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05"
Jul 17 20:28:23.848582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:23.848546    2572 scope.go:117] "RemoveContainer" containerID="3751e631c697fe4f8513e174fcd261d4d6eacb6fcb5d0013498d566a0135af47"
Jul 17 20:28:24.750735 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:24.750700    2572 generic.go:356] "Generic (PLEG): container finished" podID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerID="2b4c6adaafcad35cde419c0127579c99973d377de01455bc216508d7a3cf32ad" exitCode=6
Jul 17 20:28:24.750872 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:24.750773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" event={"ID":"a68f2f07-c17d-4579-912b-e19ca3f1dd05","Type":"ContainerDied","Data":"2b4c6adaafcad35cde419c0127579c99973d377de01455bc216508d7a3cf32ad"}
Jul 17 20:28:24.750872 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:24.750832    2572 scope.go:117] "RemoveContainer" containerID="3751e631c697fe4f8513e174fcd261d4d6eacb6fcb5d0013498d566a0135af47"
Jul 17 20:28:24.751196 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:24.751181    2572 scope.go:117] "RemoveContainer" containerID="2b4c6adaafcad35cde419c0127579c99973d377de01455bc216508d7a3cf32ad"
Jul 17 20:28:24.751378 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:24.751359    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz_odh-ai-gateway-infra(a68f2f07-c17d-4579-912b-e19ca3f1dd05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05"
Jul 17 20:28:25.775778 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:25.775748    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"]
Jul 17 20:28:25.905204 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:25.905181    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"
Jul 17 20:28:26.025660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.025625    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68f2f07-c17d-4579-912b-e19ca3f1dd05-maas-api-service-ca\") pod \"a68f2f07-c17d-4579-912b-e19ca3f1dd05\" (UID: \"a68f2f07-c17d-4579-912b-e19ca3f1dd05\") "
Jul 17 20:28:26.026063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.025996    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68f2f07-c17d-4579-912b-e19ca3f1dd05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a68f2f07-c17d-4579-912b-e19ca3f1dd05" (UID: "a68f2f07-c17d-4579-912b-e19ca3f1dd05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:28:26.127272 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.127226    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68f2f07-c17d-4579-912b-e19ca3f1dd05-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:28:26.761121 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.761091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz" event={"ID":"a68f2f07-c17d-4579-912b-e19ca3f1dd05","Type":"ContainerDied","Data":"e7015e87a406c7c5244aa82c9a6489213932a2bc60543aa83a62d5175f81c524"}
Jul 17 20:28:26.761289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.761128    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"
Jul 17 20:28:26.761289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.761159    2572 scope.go:117] "RemoveContainer" containerID="2b4c6adaafcad35cde419c0127579c99973d377de01455bc216508d7a3cf32ad"
Jul 17 20:28:26.782613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.782589    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"]
Jul 17 20:28:26.792552 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:26.788684    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-x7bhz"]
Jul 17 20:28:27.852853 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:27.852825    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" path="/var/lib/kubelet/pods/a68f2f07-c17d-4579-912b-e19ca3f1dd05/volumes"
Jul 17 20:28:36.831388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831351    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"]
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831634    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831644    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831657    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831663    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831679    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831684    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831725    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831733    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831740    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="af26dcc6-5596-4174-96d4-de1d635008ee" containerName="revoke-keys"
Jul 17 20:28:36.831850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.831747    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9945c2-6d07-4618-b62e-ec2d1a1b9da5" containerName="revoke-keys"
Jul 17 20:28:36.836101 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.836073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"
Jul 17 20:28:36.838246 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.838226    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:28:36.838246 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.838236    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:28:36.841812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:36.841778    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"]
Jul 17 20:28:37.013609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.013574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efdfd78-af40-43a9-a1de-6cefdb67e2c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-smt89\" (UID: \"3efdfd78-af40-43a9-a1de-6cefdb67e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"
Jul 17 20:28:37.114506 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.114425    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efdfd78-af40-43a9-a1de-6cefdb67e2c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-smt89\" (UID: \"3efdfd78-af40-43a9-a1de-6cefdb67e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"
Jul 17 20:28:37.115008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.114990    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efdfd78-af40-43a9-a1de-6cefdb67e2c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-smt89\" (UID: \"3efdfd78-af40-43a9-a1de-6cefdb67e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"
Jul 17 20:28:37.147616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.147589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"
Jul 17 20:28:37.274287 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.274263    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"]
Jul 17 20:28:37.276714 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:28:37.276684    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efdfd78_af40_43a9_a1de_6cefdb67e2c8.slice/crio-9368bc9aaf9bc4cca337cbf74dcefc992d4a6a51d0cf0c3aacf443797dda8303 WatchSource:0}: Error finding container 9368bc9aaf9bc4cca337cbf74dcefc992d4a6a51d0cf0c3aacf443797dda8303: Status 404 returned error can't find the container with id 9368bc9aaf9bc4cca337cbf74dcefc992d4a6a51d0cf0c3aacf443797dda8303
Jul 17 20:28:37.805601 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.805570    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" event={"ID":"3efdfd78-af40-43a9-a1de-6cefdb67e2c8","Type":"ContainerStarted","Data":"cf5ca2f458472f34fcc5aca37391a03505818a4df1bba27dcb9cf188faa20d4d"}
Jul 17 20:28:37.805775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.805618    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" event={"ID":"3efdfd78-af40-43a9-a1de-6cefdb67e2c8","Type":"ContainerStarted","Data":"9368bc9aaf9bc4cca337cbf74dcefc992d4a6a51d0cf0c3aacf443797dda8303"}
Jul 17 20:28:37.822465 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:37.822418    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" podStartSLOduration=1.822405232 podStartE2EDuration="1.822405232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:28:37.821079878 +0000 UTC m=+1952.515056025" watchObservedRunningTime="2026-07-17 20:28:37.822405232 +0000 UTC m=+1952.516381377"
Jul 17 20:28:38.811192 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:38.811159    2572 generic.go:356] "Generic (PLEG): container finished" podID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerID="cf5ca2f458472f34fcc5aca37391a03505818a4df1bba27dcb9cf188faa20d4d" exitCode=7
Jul 17 20:28:38.811641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:38.811252    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" event={"ID":"3efdfd78-af40-43a9-a1de-6cefdb67e2c8","Type":"ContainerDied","Data":"cf5ca2f458472f34fcc5aca37391a03505818a4df1bba27dcb9cf188faa20d4d"}
Jul 17 20:28:38.811641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:38.811466    2572 scope.go:117] "RemoveContainer" containerID="cf5ca2f458472f34fcc5aca37391a03505818a4df1bba27dcb9cf188faa20d4d"
Jul 17 20:28:39.816831 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:39.816732    2572 generic.go:356] "Generic (PLEG): container finished" podID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerID="f95adf5d0399768c59893a79f01c952436690e2efa58ff3f04547be44d0d5600" exitCode=7
Jul 17 20:28:39.816831 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:39.816800    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" event={"ID":"3efdfd78-af40-43a9-a1de-6cefdb67e2c8","Type":"ContainerDied","Data":"f95adf5d0399768c59893a79f01c952436690e2efa58ff3f04547be44d0d5600"}
Jul 17 20:28:39.817317 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:39.816853    2572 scope.go:117] "RemoveContainer" containerID="cf5ca2f458472f34fcc5aca37391a03505818a4df1bba27dcb9cf188faa20d4d"
Jul 17 20:28:39.817317 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:39.817016    2572 scope.go:117] "RemoveContainer" containerID="f95adf5d0399768c59893a79f01c952436690e2efa58ff3f04547be44d0d5600"
Jul 17 20:28:39.817317 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:39.817260    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-smt89_odh-ai-gateway-infra(3efdfd78-af40-43a9-a1de-6cefdb67e2c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8"
Jul 17 20:28:40.602812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.602778    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"]
Jul 17 20:28:40.603140 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.603128    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:40.603190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.603142    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:40.603224 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.603197    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68f2f07-c17d-4579-912b-e19ca3f1dd05" containerName="revoke-keys"
Jul 17 20:28:40.606823 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.606807    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"
Jul 17 20:28:40.612759 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.612726    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"]
Jul 17 20:28:40.745308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.745272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceeaae54-01e8-43d4-913c-31307541abc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8j45n\" (UID: \"ceeaae54-01e8-43d4-913c-31307541abc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"
Jul 17 20:28:40.822781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.822754    2572 scope.go:117] "RemoveContainer" containerID="f95adf5d0399768c59893a79f01c952436690e2efa58ff3f04547be44d0d5600"
Jul 17 20:28:40.823163 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:40.822918    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-smt89_odh-ai-gateway-infra(3efdfd78-af40-43a9-a1de-6cefdb67e2c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8"
Jul 17 20:28:40.845607 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.845582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceeaae54-01e8-43d4-913c-31307541abc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8j45n\" (UID: \"ceeaae54-01e8-43d4-913c-31307541abc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"
Jul 17 20:28:40.846166 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.846148    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceeaae54-01e8-43d4-913c-31307541abc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8j45n\" (UID: \"ceeaae54-01e8-43d4-913c-31307541abc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"
Jul 17 20:28:40.925756 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:40.925682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"
Jul 17 20:28:41.050794 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:41.050771    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"]
Jul 17 20:28:41.053034 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:28:41.053009    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeaae54_01e8_43d4_913c_31307541abc4.slice/crio-5a0824961a49233eb4fb023ea5cf814d4c12de2d4433a64afbbf06cb5629bf3b WatchSource:0}: Error finding container 5a0824961a49233eb4fb023ea5cf814d4c12de2d4433a64afbbf06cb5629bf3b: Status 404 returned error can't find the container with id 5a0824961a49233eb4fb023ea5cf814d4c12de2d4433a64afbbf06cb5629bf3b
Jul 17 20:28:41.827792 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:41.827756    2572 generic.go:356] "Generic (PLEG): container finished" podID="ceeaae54-01e8-43d4-913c-31307541abc4" containerID="9abf1ad6beafea179517a7b801d6360ceebf87768dc797be3d82b66b930a52fa" exitCode=6
Jul 17 20:28:41.828187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:41.827813    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" event={"ID":"ceeaae54-01e8-43d4-913c-31307541abc4","Type":"ContainerDied","Data":"9abf1ad6beafea179517a7b801d6360ceebf87768dc797be3d82b66b930a52fa"}
Jul 17 20:28:41.828187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:41.827863    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" event={"ID":"ceeaae54-01e8-43d4-913c-31307541abc4","Type":"ContainerStarted","Data":"5a0824961a49233eb4fb023ea5cf814d4c12de2d4433a64afbbf06cb5629bf3b"}
Jul 17 20:28:41.828187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:41.828076    2572 scope.go:117] "RemoveContainer" containerID="9abf1ad6beafea179517a7b801d6360ceebf87768dc797be3d82b66b930a52fa"
Jul 17 20:28:42.833629 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:42.833546    2572 generic.go:356] "Generic (PLEG): container finished" podID="ceeaae54-01e8-43d4-913c-31307541abc4" containerID="19e5f39004869832cfd99d1b92224a104c4e82ae6bd32922e79675c7b65008e1" exitCode=6
Jul 17 20:28:42.834040 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:42.833641    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" event={"ID":"ceeaae54-01e8-43d4-913c-31307541abc4","Type":"ContainerDied","Data":"19e5f39004869832cfd99d1b92224a104c4e82ae6bd32922e79675c7b65008e1"}
Jul 17 20:28:42.834040 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:42.833709    2572 scope.go:117] "RemoveContainer" containerID="9abf1ad6beafea179517a7b801d6360ceebf87768dc797be3d82b66b930a52fa"
Jul 17 20:28:42.834040 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:42.833790    2572 scope.go:117] "RemoveContainer" containerID="19e5f39004869832cfd99d1b92224a104c4e82ae6bd32922e79675c7b65008e1"
Jul 17 20:28:42.834040 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:42.833994    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8j45n_odh-ai-gateway-infra(ceeaae54-01e8-43d4-913c-31307541abc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" podUID="ceeaae54-01e8-43d4-913c-31307541abc4"
Jul 17 20:28:43.839258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:43.839232    2572 scope.go:117] "RemoveContainer" containerID="19e5f39004869832cfd99d1b92224a104c4e82ae6bd32922e79675c7b65008e1"
Jul 17 20:28:43.839630 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:43.839412    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8j45n_odh-ai-gateway-infra(ceeaae54-01e8-43d4-913c-31307541abc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" podUID="ceeaae54-01e8-43d4-913c-31307541abc4"
Jul 17 20:28:53.847906 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:53.847869    2572 scope.go:117] "RemoveContainer" containerID="f95adf5d0399768c59893a79f01c952436690e2efa58ff3f04547be44d0d5600"
Jul 17 20:28:54.891075 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:54.891029    2572 generic.go:356] "Generic (PLEG): container finished" podID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerID="edd731d3d9321c1cfd4187d8441f82633b00126a9f0fa42c465c229f6a566cc0" exitCode=7
Jul 17 20:28:54.891501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:54.891101    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" event={"ID":"3efdfd78-af40-43a9-a1de-6cefdb67e2c8","Type":"ContainerDied","Data":"edd731d3d9321c1cfd4187d8441f82633b00126a9f0fa42c465c229f6a566cc0"}
Jul 17 20:28:54.891501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:54.891166    2572 scope.go:117] "RemoveContainer" containerID="f95adf5d0399768c59893a79f01c952436690e2efa58ff3f04547be44d0d5600"
Jul 17 20:28:54.891501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:54.891431    2572 scope.go:117] "RemoveContainer" containerID="edd731d3d9321c1cfd4187d8441f82633b00126a9f0fa42c465c229f6a566cc0"
Jul 17 20:28:54.891669 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:54.891647    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-smt89_odh-ai-gateway-infra(3efdfd78-af40-43a9-a1de-6cefdb67e2c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8"
Jul 17 20:28:55.939375 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:55.939337    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"]
Jul 17 20:28:56.068393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.068371    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"
Jul 17 20:28:56.260899 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.260872    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efdfd78-af40-43a9-a1de-6cefdb67e2c8-maas-api-service-ca\") pod \"3efdfd78-af40-43a9-a1de-6cefdb67e2c8\" (UID: \"3efdfd78-af40-43a9-a1de-6cefdb67e2c8\") "
Jul 17 20:28:56.261253 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.261232    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efdfd78-af40-43a9-a1de-6cefdb67e2c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3efdfd78-af40-43a9-a1de-6cefdb67e2c8" (UID: "3efdfd78-af40-43a9-a1de-6cefdb67e2c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:28:56.361397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.361357    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efdfd78-af40-43a9-a1de-6cefdb67e2c8-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:28:56.902383 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.902352    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"
Jul 17 20:28:56.902574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.902356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89" event={"ID":"3efdfd78-af40-43a9-a1de-6cefdb67e2c8","Type":"ContainerDied","Data":"9368bc9aaf9bc4cca337cbf74dcefc992d4a6a51d0cf0c3aacf443797dda8303"}
Jul 17 20:28:56.902574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.902477    2572 scope.go:117] "RemoveContainer" containerID="edd731d3d9321c1cfd4187d8441f82633b00126a9f0fa42c465c229f6a566cc0"
Jul 17 20:28:56.923108 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.923080    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"]
Jul 17 20:28:56.926478 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:56.926457    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-smt89"]
Jul 17 20:28:57.852882 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:57.852806    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" path="/var/lib/kubelet/pods/3efdfd78-af40-43a9-a1de-6cefdb67e2c8/volumes"
Jul 17 20:28:58.624478 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624437    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"]
Jul 17 20:28:58.624794 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624782    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:28:58.624841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624796    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:28:58.624841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624813    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:28:58.624841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624818    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:28:58.624948 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624867    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:28:58.624948 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624875    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:28:58.624948 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.624882    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:28:58.629029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.629013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"
Jul 17 20:28:58.634584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.634519    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"]
Jul 17 20:28:58.674414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.674378    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c65-e7d6-4e7b-ac8d-907558023b81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw\" (UID: \"beb82c65-e7d6-4e7b-ac8d-907558023b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"
Jul 17 20:28:58.775112 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.775074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c65-e7d6-4e7b-ac8d-907558023b81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw\" (UID: \"beb82c65-e7d6-4e7b-ac8d-907558023b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"
Jul 17 20:28:58.775651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.775632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c65-e7d6-4e7b-ac8d-907558023b81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw\" (UID: \"beb82c65-e7d6-4e7b-ac8d-907558023b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"
Jul 17 20:28:58.848440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.848414    2572 scope.go:117] "RemoveContainer" containerID="19e5f39004869832cfd99d1b92224a104c4e82ae6bd32922e79675c7b65008e1"
Jul 17 20:28:58.940135 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:58.940105    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"
Jul 17 20:28:59.141012 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.140893    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"]
Jul 17 20:28:59.143803 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:28:59.143764    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb82c65_e7d6_4e7b_ac8d_907558023b81.slice/crio-78355851fc8ada78fd51455cc5c1383f7cd9e86439aed73df3713c731d8e396d WatchSource:0}: Error finding container 78355851fc8ada78fd51455cc5c1383f7cd9e86439aed73df3713c731d8e396d: Status 404 returned error can't find the container with id 78355851fc8ada78fd51455cc5c1383f7cd9e86439aed73df3713c731d8e396d
Jul 17 20:28:59.916880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.916844    2572 generic.go:356] "Generic (PLEG): container finished" podID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerID="b576688274bfe82d2cf8c08fe11dd05a8a6125cf019f75aed1b72d9135945427" exitCode=6
Jul 17 20:28:59.917077 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.916930    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" event={"ID":"beb82c65-e7d6-4e7b-ac8d-907558023b81","Type":"ContainerDied","Data":"b576688274bfe82d2cf8c08fe11dd05a8a6125cf019f75aed1b72d9135945427"}
Jul 17 20:28:59.917077 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.916987    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" event={"ID":"beb82c65-e7d6-4e7b-ac8d-907558023b81","Type":"ContainerStarted","Data":"78355851fc8ada78fd51455cc5c1383f7cd9e86439aed73df3713c731d8e396d"}
Jul 17 20:28:59.917220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.917161    2572 scope.go:117] "RemoveContainer" containerID="b576688274bfe82d2cf8c08fe11dd05a8a6125cf019f75aed1b72d9135945427"
Jul 17 20:28:59.918788 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.918766    2572 generic.go:356] "Generic (PLEG): container finished" podID="ceeaae54-01e8-43d4-913c-31307541abc4" containerID="2f4332b8e636b0fcc23bc5994dc77ca5c4b26392534039f0ffd7329a9dbbea3f" exitCode=6
Jul 17 20:28:59.918869 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.918805    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" event={"ID":"ceeaae54-01e8-43d4-913c-31307541abc4","Type":"ContainerDied","Data":"2f4332b8e636b0fcc23bc5994dc77ca5c4b26392534039f0ffd7329a9dbbea3f"}
Jul 17 20:28:59.918869 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.918845    2572 scope.go:117] "RemoveContainer" containerID="19e5f39004869832cfd99d1b92224a104c4e82ae6bd32922e79675c7b65008e1"
Jul 17 20:28:59.919090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:28:59.919075    2572 scope.go:117] "RemoveContainer" containerID="2f4332b8e636b0fcc23bc5994dc77ca5c4b26392534039f0ffd7329a9dbbea3f"
Jul 17 20:28:59.919257 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:28:59.919240    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8j45n_odh-ai-gateway-infra(ceeaae54-01e8-43d4-913c-31307541abc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" podUID="ceeaae54-01e8-43d4-913c-31307541abc4"
Jul 17 20:29:00.925809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:00.925716    2572 generic.go:356] "Generic (PLEG): container finished" podID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerID="18c49503d79da0e04c1c495f8657b5ae412b91c8edcc2d9b4327c4f1caff5c54" exitCode=6
Jul 17 20:29:00.925809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:00.925796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" event={"ID":"beb82c65-e7d6-4e7b-ac8d-907558023b81","Type":"ContainerDied","Data":"18c49503d79da0e04c1c495f8657b5ae412b91c8edcc2d9b4327c4f1caff5c54"}
Jul 17 20:29:00.926237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:00.925848    2572 scope.go:117] "RemoveContainer" containerID="b576688274bfe82d2cf8c08fe11dd05a8a6125cf019f75aed1b72d9135945427"
Jul 17 20:29:00.926237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:00.926097    2572 scope.go:117] "RemoveContainer" containerID="18c49503d79da0e04c1c495f8657b5ae412b91c8edcc2d9b4327c4f1caff5c54"
Jul 17 20:29:00.926338 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:00.926294    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw_odh-ai-gateway-infra(beb82c65-e7d6-4e7b-ac8d-907558023b81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81"
Jul 17 20:29:00.954163 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:00.954132    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"]
Jul 17 20:29:01.081909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.081886    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"
Jul 17 20:29:01.088437 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.088417    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceeaae54-01e8-43d4-913c-31307541abc4-maas-api-service-ca\") pod \"ceeaae54-01e8-43d4-913c-31307541abc4\" (UID: \"ceeaae54-01e8-43d4-913c-31307541abc4\") "
Jul 17 20:29:01.088718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.088697    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceeaae54-01e8-43d4-913c-31307541abc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ceeaae54-01e8-43d4-913c-31307541abc4" (UID: "ceeaae54-01e8-43d4-913c-31307541abc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:29:01.189010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.188931    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceeaae54-01e8-43d4-913c-31307541abc4-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:29:01.931497 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.931468    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"
Jul 17 20:29:01.931936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.931498    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n" event={"ID":"ceeaae54-01e8-43d4-913c-31307541abc4","Type":"ContainerDied","Data":"5a0824961a49233eb4fb023ea5cf814d4c12de2d4433a64afbbf06cb5629bf3b"}
Jul 17 20:29:01.931936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.931564    2572 scope.go:117] "RemoveContainer" containerID="2f4332b8e636b0fcc23bc5994dc77ca5c4b26392534039f0ffd7329a9dbbea3f"
Jul 17 20:29:01.933436 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.933406    2572 scope.go:117] "RemoveContainer" containerID="18c49503d79da0e04c1c495f8657b5ae412b91c8edcc2d9b4327c4f1caff5c54"
Jul 17 20:29:01.933662 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:01.933620    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw_odh-ai-gateway-infra(beb82c65-e7d6-4e7b-ac8d-907558023b81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81"
Jul 17 20:29:01.966680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.966645    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"]
Jul 17 20:29:01.972924 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:01.972897    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8j45n"]
Jul 17 20:29:03.854257 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:03.854223    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" path="/var/lib/kubelet/pods/ceeaae54-01e8-43d4-913c-31307541abc4/volumes"
Jul 17 20:29:13.848357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:13.848326    2572 scope.go:117] "RemoveContainer" containerID="18c49503d79da0e04c1c495f8657b5ae412b91c8edcc2d9b4327c4f1caff5c54"
Jul 17 20:29:14.987468 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:14.987439    2572 generic.go:356] "Generic (PLEG): container finished" podID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerID="a79793bf3a7f06f9af152a132f906bec5aae0e00021f82528084d2404d4a5d86" exitCode=6
Jul 17 20:29:14.987883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:14.987514    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" event={"ID":"beb82c65-e7d6-4e7b-ac8d-907558023b81","Type":"ContainerDied","Data":"a79793bf3a7f06f9af152a132f906bec5aae0e00021f82528084d2404d4a5d86"}
Jul 17 20:29:14.987883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:14.987575    2572 scope.go:117] "RemoveContainer" containerID="18c49503d79da0e04c1c495f8657b5ae412b91c8edcc2d9b4327c4f1caff5c54"
Jul 17 20:29:14.987883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:14.987841    2572 scope.go:117] "RemoveContainer" containerID="a79793bf3a7f06f9af152a132f906bec5aae0e00021f82528084d2404d4a5d86"
Jul 17 20:29:14.988071 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:14.988038    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw_odh-ai-gateway-infra(beb82c65-e7d6-4e7b-ac8d-907558023b81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81"
Jul 17 20:29:16.012268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.012231    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"]
Jul 17 20:29:16.140324 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.140299    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"
Jul 17 20:29:16.205657 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.205622    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c65-e7d6-4e7b-ac8d-907558023b81-maas-api-service-ca\") pod \"beb82c65-e7d6-4e7b-ac8d-907558023b81\" (UID: \"beb82c65-e7d6-4e7b-ac8d-907558023b81\") "
Jul 17 20:29:16.205996 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.205969    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb82c65-e7d6-4e7b-ac8d-907558023b81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "beb82c65-e7d6-4e7b-ac8d-907558023b81" (UID: "beb82c65-e7d6-4e7b-ac8d-907558023b81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:29:16.306838 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.306762    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c65-e7d6-4e7b-ac8d-907558023b81-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:29:16.999122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.999091    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"
Jul 17 20:29:16.999122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.999101    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw" event={"ID":"beb82c65-e7d6-4e7b-ac8d-907558023b81","Type":"ContainerDied","Data":"78355851fc8ada78fd51455cc5c1383f7cd9e86439aed73df3713c731d8e396d"}
Jul 17 20:29:16.999378 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:16.999161    2572 scope.go:117] "RemoveContainer" containerID="a79793bf3a7f06f9af152a132f906bec5aae0e00021f82528084d2404d4a5d86"
Jul 17 20:29:17.019003 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:17.018979    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"]
Jul 17 20:29:17.032466 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:17.032442    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-zkgtw"]
Jul 17 20:29:17.853782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:17.853747    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" path="/var/lib/kubelet/pods/beb82c65-e7d6-4e7b-ac8d-907558023b81/volumes"
Jul 17 20:29:36.878178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878141    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"]
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878440    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878451    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878461    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878466    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878476    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878482    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3efdfd78-af40-43a9-a1de-6cefdb67e2c8" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878493    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878498    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878508    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878512    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878569    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878578    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878584    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878590    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:36.878634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.878595    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:36.882800 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.882784    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"
Jul 17 20:29:36.886558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.886526    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:29:36.886651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.886587    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:29:36.892502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.892480    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"]
Jul 17 20:29:36.972273 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:36.972232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4ff6e4-386e-46e2-a79b-373d638ea8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-66577\" (UID: \"ac4ff6e4-386e-46e2-a79b-373d638ea8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"
Jul 17 20:29:37.072668 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:37.072630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4ff6e4-386e-46e2-a79b-373d638ea8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-66577\" (UID: \"ac4ff6e4-386e-46e2-a79b-373d638ea8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"
Jul 17 20:29:37.073276 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:37.073259    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4ff6e4-386e-46e2-a79b-373d638ea8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-66577\" (UID: \"ac4ff6e4-386e-46e2-a79b-373d638ea8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"
Jul 17 20:29:37.193548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:37.193464    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"
Jul 17 20:29:37.324333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:37.324246    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"]
Jul 17 20:29:37.326626 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:29:37.326588    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4ff6e4_386e_46e2_a79b_373d638ea8ff.slice/crio-bc8b88aa1e6e6edaac595b501481f6c7f928c4034d4e1eb51cd3dde5142da869 WatchSource:0}: Error finding container bc8b88aa1e6e6edaac595b501481f6c7f928c4034d4e1eb51cd3dde5142da869: Status 404 returned error can't find the container with id bc8b88aa1e6e6edaac595b501481f6c7f928c4034d4e1eb51cd3dde5142da869
Jul 17 20:29:38.088295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:38.088254    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerID="f2e84ad384634dbe1f7605f0fce5186eef8fa7bf147e1d0ad4948448de1dcee3" exitCode=7
Jul 17 20:29:38.088705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:38.088313    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" event={"ID":"ac4ff6e4-386e-46e2-a79b-373d638ea8ff","Type":"ContainerDied","Data":"f2e84ad384634dbe1f7605f0fce5186eef8fa7bf147e1d0ad4948448de1dcee3"}
Jul 17 20:29:38.088705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:38.088363    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" event={"ID":"ac4ff6e4-386e-46e2-a79b-373d638ea8ff","Type":"ContainerStarted","Data":"bc8b88aa1e6e6edaac595b501481f6c7f928c4034d4e1eb51cd3dde5142da869"}
Jul 17 20:29:38.088705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:38.088532    2572 scope.go:117] "RemoveContainer" containerID="f2e84ad384634dbe1f7605f0fce5186eef8fa7bf147e1d0ad4948448de1dcee3"
Jul 17 20:29:39.094556 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:39.094458    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerID="ae78b56f3605ead326da73f2f2d9804cfc85bb9d3f7e005fec9522e91c97d159" exitCode=7
Jul 17 20:29:39.094556 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:39.094502    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" event={"ID":"ac4ff6e4-386e-46e2-a79b-373d638ea8ff","Type":"ContainerDied","Data":"ae78b56f3605ead326da73f2f2d9804cfc85bb9d3f7e005fec9522e91c97d159"}
Jul 17 20:29:39.094556 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:39.094555    2572 scope.go:117] "RemoveContainer" containerID="f2e84ad384634dbe1f7605f0fce5186eef8fa7bf147e1d0ad4948448de1dcee3"
Jul 17 20:29:39.095071 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:39.094719    2572 scope.go:117] "RemoveContainer" containerID="ae78b56f3605ead326da73f2f2d9804cfc85bb9d3f7e005fec9522e91c97d159"
Jul 17 20:29:39.095071 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:39.094945    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-66577_odh-ai-gateway-infra(ac4ff6e4-386e-46e2-a79b-373d638ea8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff"
Jul 17 20:29:40.100834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.100804    2572 scope.go:117] "RemoveContainer" containerID="ae78b56f3605ead326da73f2f2d9804cfc85bb9d3f7e005fec9522e91c97d159"
Jul 17 20:29:40.101283 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:40.101035    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-66577_odh-ai-gateway-infra(ac4ff6e4-386e-46e2-a79b-373d638ea8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff"
Jul 17 20:29:40.655616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.655580    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"]
Jul 17 20:29:40.655921 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.655909    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:40.655964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.655924    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:40.655964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.655939    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:40.655964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.655946    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceeaae54-01e8-43d4-913c-31307541abc4" containerName="revoke-keys"
Jul 17 20:29:40.656079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.655990    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb82c65-e7d6-4e7b-ac8d-907558023b81" containerName="revoke-keys"
Jul 17 20:29:40.659057 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.659028    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"
Jul 17 20:29:40.667790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.667761    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"]
Jul 17 20:29:40.806762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.806710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468fa0f-a251-4424-a991-26df34baa2cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8pzxv\" (UID: \"6468fa0f-a251-4424-a991-26df34baa2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"
Jul 17 20:29:40.907388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.907291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468fa0f-a251-4424-a991-26df34baa2cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8pzxv\" (UID: \"6468fa0f-a251-4424-a991-26df34baa2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"
Jul 17 20:29:40.907919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.907900    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468fa0f-a251-4424-a991-26df34baa2cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8pzxv\" (UID: \"6468fa0f-a251-4424-a991-26df34baa2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"
Jul 17 20:29:40.970524 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:40.970491    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"
Jul 17 20:29:41.121692 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:41.121667    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"]
Jul 17 20:29:41.124096 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:29:41.124063    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6468fa0f_a251_4424_a991_26df34baa2cf.slice/crio-320ee2de051d2ee857b79c6235d114c316f4a9eb5ebbe6ffb36f183e000593d1 WatchSource:0}: Error finding container 320ee2de051d2ee857b79c6235d114c316f4a9eb5ebbe6ffb36f183e000593d1: Status 404 returned error can't find the container with id 320ee2de051d2ee857b79c6235d114c316f4a9eb5ebbe6ffb36f183e000593d1
Jul 17 20:29:42.111094 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:42.111005    2572 generic.go:356] "Generic (PLEG): container finished" podID="6468fa0f-a251-4424-a991-26df34baa2cf" containerID="3d78ab987a67e3808d8491f3751e87e69d4c4d3f42bc95f591b1b794bd57ee21" exitCode=6
Jul 17 20:29:42.111094 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:42.111080    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" event={"ID":"6468fa0f-a251-4424-a991-26df34baa2cf","Type":"ContainerDied","Data":"3d78ab987a67e3808d8491f3751e87e69d4c4d3f42bc95f591b1b794bd57ee21"}
Jul 17 20:29:42.111336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:42.111139    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" event={"ID":"6468fa0f-a251-4424-a991-26df34baa2cf","Type":"ContainerStarted","Data":"320ee2de051d2ee857b79c6235d114c316f4a9eb5ebbe6ffb36f183e000593d1"}
Jul 17 20:29:42.111336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:42.111323    2572 scope.go:117] "RemoveContainer" containerID="3d78ab987a67e3808d8491f3751e87e69d4c4d3f42bc95f591b1b794bd57ee21"
Jul 17 20:29:43.117579 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:43.117483    2572 generic.go:356] "Generic (PLEG): container finished" podID="6468fa0f-a251-4424-a991-26df34baa2cf" containerID="b355ec7c658541143459ca7fc433a147d9c718e0842372bc6e06f9e6b0242a60" exitCode=6
Jul 17 20:29:43.117979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:43.117570    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" event={"ID":"6468fa0f-a251-4424-a991-26df34baa2cf","Type":"ContainerDied","Data":"b355ec7c658541143459ca7fc433a147d9c718e0842372bc6e06f9e6b0242a60"}
Jul 17 20:29:43.117979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:43.117628    2572 scope.go:117] "RemoveContainer" containerID="3d78ab987a67e3808d8491f3751e87e69d4c4d3f42bc95f591b1b794bd57ee21"
Jul 17 20:29:43.117979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:43.117783    2572 scope.go:117] "RemoveContainer" containerID="b355ec7c658541143459ca7fc433a147d9c718e0842372bc6e06f9e6b0242a60"
Jul 17 20:29:43.118131 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:43.117998    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8pzxv_odh-ai-gateway-infra(6468fa0f-a251-4424-a991-26df34baa2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" podUID="6468fa0f-a251-4424-a991-26df34baa2cf"
Jul 17 20:29:44.123674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:44.123648    2572 scope.go:117] "RemoveContainer" containerID="b355ec7c658541143459ca7fc433a147d9c718e0842372bc6e06f9e6b0242a60"
Jul 17 20:29:44.124069 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:44.123838    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8pzxv_odh-ai-gateway-infra(6468fa0f-a251-4424-a991-26df34baa2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" podUID="6468fa0f-a251-4424-a991-26df34baa2cf"
Jul 17 20:29:48.686000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:48.685969    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"]
Jul 17 20:29:48.690459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:48.690442    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"
Jul 17 20:29:48.697506 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:48.697268    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"]
Jul 17 20:29:48.773162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:48.773125    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76038b90-5283-4c95-8e4a-6d399fc1d3b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz\" (UID: \"76038b90-5283-4c95-8e4a-6d399fc1d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"
Jul 17 20:29:48.873874 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:48.873840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76038b90-5283-4c95-8e4a-6d399fc1d3b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz\" (UID: \"76038b90-5283-4c95-8e4a-6d399fc1d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"
Jul 17 20:29:48.874429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:48.874412    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76038b90-5283-4c95-8e4a-6d399fc1d3b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz\" (UID: \"76038b90-5283-4c95-8e4a-6d399fc1d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"
Jul 17 20:29:49.001669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:49.001639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"
Jul 17 20:29:49.130391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:49.130286    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"]
Jul 17 20:29:49.132420 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:29:49.132395    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76038b90_5283_4c95_8e4a_6d399fc1d3b1.slice/crio-5fdf12f5ec1b3122674f57d421cc5ad6c364b8340601911811f88258c2ee6f70 WatchSource:0}: Error finding container 5fdf12f5ec1b3122674f57d421cc5ad6c364b8340601911811f88258c2ee6f70: Status 404 returned error can't find the container with id 5fdf12f5ec1b3122674f57d421cc5ad6c364b8340601911811f88258c2ee6f70
Jul 17 20:29:49.144396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:49.144370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" event={"ID":"76038b90-5283-4c95-8e4a-6d399fc1d3b1","Type":"ContainerStarted","Data":"5fdf12f5ec1b3122674f57d421cc5ad6c364b8340601911811f88258c2ee6f70"}
Jul 17 20:29:50.150845 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:50.150768    2572 generic.go:356] "Generic (PLEG): container finished" podID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerID="fb778e756831e07fd9decc52680beb04d77f4f245fce4bca41005d488effa4c4" exitCode=6
Jul 17 20:29:50.150845 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:50.150825    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" event={"ID":"76038b90-5283-4c95-8e4a-6d399fc1d3b1","Type":"ContainerDied","Data":"fb778e756831e07fd9decc52680beb04d77f4f245fce4bca41005d488effa4c4"}
Jul 17 20:29:50.151235 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:50.151024    2572 scope.go:117] "RemoveContainer" containerID="fb778e756831e07fd9decc52680beb04d77f4f245fce4bca41005d488effa4c4"
Jul 17 20:29:51.156620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:51.156535    2572 generic.go:356] "Generic (PLEG): container finished" podID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerID="c7ff305bb92f2b44001734cb17c895d935138d25e6d70640315408aed007fcd4" exitCode=6
Jul 17 20:29:51.157029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:51.156628    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" event={"ID":"76038b90-5283-4c95-8e4a-6d399fc1d3b1","Type":"ContainerDied","Data":"c7ff305bb92f2b44001734cb17c895d935138d25e6d70640315408aed007fcd4"}
Jul 17 20:29:51.157029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:51.156721    2572 scope.go:117] "RemoveContainer" containerID="fb778e756831e07fd9decc52680beb04d77f4f245fce4bca41005d488effa4c4"
Jul 17 20:29:51.157029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:51.156846    2572 scope.go:117] "RemoveContainer" containerID="c7ff305bb92f2b44001734cb17c895d935138d25e6d70640315408aed007fcd4"
Jul 17 20:29:51.157177 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:51.157066    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz_odh-ai-gateway-infra(76038b90-5283-4c95-8e4a-6d399fc1d3b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1"
Jul 17 20:29:52.162520 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:52.162493    2572 scope.go:117] "RemoveContainer" containerID="c7ff305bb92f2b44001734cb17c895d935138d25e6d70640315408aed007fcd4"
Jul 17 20:29:52.162925 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:52.162691    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz_odh-ai-gateway-infra(76038b90-5283-4c95-8e4a-6d399fc1d3b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1"
Jul 17 20:29:54.847894 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:54.847862    2572 scope.go:117] "RemoveContainer" containerID="ae78b56f3605ead326da73f2f2d9804cfc85bb9d3f7e005fec9522e91c97d159"
Jul 17 20:29:55.177124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:55.177017    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" event={"ID":"ac4ff6e4-386e-46e2-a79b-373d638ea8ff","Type":"ContainerStarted","Data":"febfb6ff10e4c0425356a1ca55f772cae3ea91518a29b6165457279406271356"}
Jul 17 20:29:55.193450 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:55.193394    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" podStartSLOduration=19.193376963 podStartE2EDuration="19.193376963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:29:55.192275031 +0000 UTC m=+2029.886251187" watchObservedRunningTime="2026-07-17 20:29:55.193376963 +0000 UTC m=+2029.887353109"
Jul 17 20:29:56.182718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:56.182687    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerID="febfb6ff10e4c0425356a1ca55f772cae3ea91518a29b6165457279406271356" exitCode=7
Jul 17 20:29:56.183182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:56.182766    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" event={"ID":"ac4ff6e4-386e-46e2-a79b-373d638ea8ff","Type":"ContainerDied","Data":"febfb6ff10e4c0425356a1ca55f772cae3ea91518a29b6165457279406271356"}
Jul 17 20:29:56.183182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:56.182836    2572 scope.go:117] "RemoveContainer" containerID="ae78b56f3605ead326da73f2f2d9804cfc85bb9d3f7e005fec9522e91c97d159"
Jul 17 20:29:56.183182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:56.183113    2572 scope.go:117] "RemoveContainer" containerID="febfb6ff10e4c0425356a1ca55f772cae3ea91518a29b6165457279406271356"
Jul 17 20:29:56.183379 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:29:56.183359    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-66577_odh-ai-gateway-infra(ac4ff6e4-386e-46e2-a79b-373d638ea8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff"
Jul 17 20:29:56.207725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:56.207699    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"]
Jul 17 20:29:57.324272 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:57.324246    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"
Jul 17 20:29:57.447773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:57.447688    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4ff6e4-386e-46e2-a79b-373d638ea8ff-maas-api-service-ca\") pod \"ac4ff6e4-386e-46e2-a79b-373d638ea8ff\" (UID: \"ac4ff6e4-386e-46e2-a79b-373d638ea8ff\") "
Jul 17 20:29:57.448031 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:57.448007    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4ff6e4-386e-46e2-a79b-373d638ea8ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac4ff6e4-386e-46e2-a79b-373d638ea8ff" (UID: "ac4ff6e4-386e-46e2-a79b-373d638ea8ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:29:57.549155 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:57.549107    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4ff6e4-386e-46e2-a79b-373d638ea8ff-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:29:58.195596 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:58.195496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577" event={"ID":"ac4ff6e4-386e-46e2-a79b-373d638ea8ff","Type":"ContainerDied","Data":"bc8b88aa1e6e6edaac595b501481f6c7f928c4034d4e1eb51cd3dde5142da869"}
Jul 17 20:29:58.195596 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:58.195573    2572 scope.go:117] "RemoveContainer" containerID="febfb6ff10e4c0425356a1ca55f772cae3ea91518a29b6165457279406271356"
Jul 17 20:29:58.195810 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:58.195707    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"
Jul 17 20:29:58.211762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:58.211734    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"]
Jul 17 20:29:58.218435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:58.218411    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-66577"]
Jul 17 20:29:58.848417 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:58.848387    2572 scope.go:117] "RemoveContainer" containerID="b355ec7c658541143459ca7fc433a147d9c718e0842372bc6e06f9e6b0242a60"
Jul 17 20:29:59.202511 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:59.202479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" event={"ID":"6468fa0f-a251-4424-a991-26df34baa2cf","Type":"ContainerStarted","Data":"7dc8f3ad510ac4b8e4ee9e84bf073dfbabd117518a99f80191a7adf6941d2a36"}
Jul 17 20:29:59.226588 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:59.226518    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" podStartSLOduration=19.226494679 podStartE2EDuration="19.226494679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:29:59.224446638 +0000 UTC m=+2033.918422786" watchObservedRunningTime="2026-07-17 20:29:59.226494679 +0000 UTC m=+2033.920470826"
Jul 17 20:29:59.853890 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:29:59.853807    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" path="/var/lib/kubelet/pods/ac4ff6e4-386e-46e2-a79b-373d638ea8ff/volumes"
Jul 17 20:30:00.181746 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.181665    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"]
Jul 17 20:30:00.182165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.182146    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.182220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.182169    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.182220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.182204    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.182220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.182213    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.182322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.182286    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.182322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.182300    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.185519 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.185500    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc"]
Jul 17 20:30:00.185648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.185629    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"
Jul 17 20:30:00.185897 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.185882    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.185931 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.185900    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.185989 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.185979    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4ff6e4-386e-46e2-a79b-373d638ea8ff" containerName="revoke-keys"
Jul 17 20:30:00.188499 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.188485    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc"
Jul 17 20:30:00.195312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.195298    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"
Jul 17 20:30:00.202628 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.202603    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc"
Jul 17 20:30:00.204912 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.204890    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc"]
Jul 17 20:30:00.213505 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.212900    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"]
Jul 17 20:30:00.216292 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.216253    2572 generic.go:356] "Generic (PLEG): container finished" podID="6468fa0f-a251-4424-a991-26df34baa2cf" containerID="7dc8f3ad510ac4b8e4ee9e84bf073dfbabd117518a99f80191a7adf6941d2a36" exitCode=6
Jul 17 20:30:00.216388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.216370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" event={"ID":"6468fa0f-a251-4424-a991-26df34baa2cf","Type":"ContainerDied","Data":"7dc8f3ad510ac4b8e4ee9e84bf073dfbabd117518a99f80191a7adf6941d2a36"}
Jul 17 20:30:00.216446 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.216423    2572 scope.go:117] "RemoveContainer" containerID="b355ec7c658541143459ca7fc433a147d9c718e0842372bc6e06f9e6b0242a60"
Jul 17 20:30:00.217092 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.216750    2572 scope.go:117] "RemoveContainer" containerID="7dc8f3ad510ac4b8e4ee9e84bf073dfbabd117518a99f80191a7adf6941d2a36"
Jul 17 20:30:00.217092 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:00.216972    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8pzxv_odh-ai-gateway-infra(6468fa0f-a251-4424-a991-26df34baa2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" podUID="6468fa0f-a251-4424-a991-26df34baa2cf"
Jul 17 20:30:00.245738 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.245695    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"]
Jul 17 20:30:00.364008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.363834    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc"]
Jul 17 20:30:00.366550 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:30:00.366512    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f320eb5_cac2_48a5_a8bc_3c829238e69c.slice/crio-3e5ecd70ae0227a6f70a53538a68fc216a1191cf9d1aae7f2a1aa76f646faa57 WatchSource:0}: Error finding container 3e5ecd70ae0227a6f70a53538a68fc216a1191cf9d1aae7f2a1aa76f646faa57: Status 404 returned error can't find the container with id 3e5ecd70ae0227a6f70a53538a68fc216a1191cf9d1aae7f2a1aa76f646faa57
Jul 17 20:30:00.381711 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:00.381686    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"]
Jul 17 20:30:00.383481 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:30:00.383456    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc5cb1a_007d_43c5_97ab_282c74f4fd71.slice/crio-ca5b6826f05106677b8143023748ccfad569082d390d40b76e909ff8077dcc21 WatchSource:0}: Error finding container ca5b6826f05106677b8143023748ccfad569082d390d40b76e909ff8077dcc21: Status 404 returned error can't find the container with id ca5b6826f05106677b8143023748ccfad569082d390d40b76e909ff8077dcc21
Jul 17 20:30:01.221899 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.221852    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc" event={"ID":"8f320eb5-cac2-48a5-a8bc-3c829238e69c","Type":"ContainerStarted","Data":"296610eeec62c6e2ebf6507961b257bed2713c0245fe54fbe8d6e5205c4d88cf"}
Jul 17 20:30:01.222327 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.221956    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc" event={"ID":"8f320eb5-cac2-48a5-a8bc-3c829238e69c","Type":"ContainerStarted","Data":"3e5ecd70ae0227a6f70a53538a68fc216a1191cf9d1aae7f2a1aa76f646faa57"}
Jul 17 20:30:01.223251 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.223231    2572 generic.go:356] "Generic (PLEG): container finished" podID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerID="55ebe470436fcd974100de0d5471c0627b35a0ed04c1c2393c3e58699b89fa5d" exitCode=7
Jul 17 20:30:01.223343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.223317    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" event={"ID":"0fc5cb1a-007d-43c5-97ab-282c74f4fd71","Type":"ContainerDied","Data":"55ebe470436fcd974100de0d5471c0627b35a0ed04c1c2393c3e58699b89fa5d"}
Jul 17 20:30:01.223405 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.223370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" event={"ID":"0fc5cb1a-007d-43c5-97ab-282c74f4fd71","Type":"ContainerStarted","Data":"ca5b6826f05106677b8143023748ccfad569082d390d40b76e909ff8077dcc21"}
Jul 17 20:30:01.223463 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.223434    2572 scope.go:117] "RemoveContainer" containerID="55ebe470436fcd974100de0d5471c0627b35a0ed04c1c2393c3e58699b89fa5d"
Jul 17 20:30:01.278327 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.278274    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc" podStartSLOduration=1.278254952 podStartE2EDuration="1.278254952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:30:01.246642526 +0000 UTC m=+2035.940618672" watchObservedRunningTime="2026-07-17 20:30:01.278254952 +0000 UTC m=+2035.972231097"
Jul 17 20:30:01.396199 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.396175    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"
Jul 17 20:30:01.479907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.479876    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468fa0f-a251-4424-a991-26df34baa2cf-maas-api-service-ca\") pod \"6468fa0f-a251-4424-a991-26df34baa2cf\" (UID: \"6468fa0f-a251-4424-a991-26df34baa2cf\") "
Jul 17 20:30:01.480242 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.480221    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6468fa0f-a251-4424-a991-26df34baa2cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6468fa0f-a251-4424-a991-26df34baa2cf" (UID: "6468fa0f-a251-4424-a991-26df34baa2cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:30:01.581091 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:01.581065    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468fa0f-a251-4424-a991-26df34baa2cf-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:30:02.230376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.230349    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"
Jul 17 20:30:02.230828 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.230384    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv" event={"ID":"6468fa0f-a251-4424-a991-26df34baa2cf","Type":"ContainerDied","Data":"320ee2de051d2ee857b79c6235d114c316f4a9eb5ebbe6ffb36f183e000593d1"}
Jul 17 20:30:02.230828 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.230439    2572 scope.go:117] "RemoveContainer" containerID="7dc8f3ad510ac4b8e4ee9e84bf073dfbabd117518a99f80191a7adf6941d2a36"
Jul 17 20:30:02.231973 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.231948    2572 generic.go:356] "Generic (PLEG): container finished" podID="8f320eb5-cac2-48a5-a8bc-3c829238e69c" containerID="296610eeec62c6e2ebf6507961b257bed2713c0245fe54fbe8d6e5205c4d88cf" exitCode=0
Jul 17 20:30:02.232101 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.232027    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc" event={"ID":"8f320eb5-cac2-48a5-a8bc-3c829238e69c","Type":"ContainerDied","Data":"296610eeec62c6e2ebf6507961b257bed2713c0245fe54fbe8d6e5205c4d88cf"}
Jul 17 20:30:02.234033 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.234014    2572 generic.go:356] "Generic (PLEG): container finished" podID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerID="e44f5f4e25f717e3760dcb048f9eea69578ad17d90c7f049f4b001bf84109a63" exitCode=7
Jul 17 20:30:02.234136 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.234073    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" event={"ID":"0fc5cb1a-007d-43c5-97ab-282c74f4fd71","Type":"ContainerDied","Data":"e44f5f4e25f717e3760dcb048f9eea69578ad17d90c7f049f4b001bf84109a63"}
Jul 17 20:30:02.234238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.234223    2572 scope.go:117] "RemoveContainer" containerID="e44f5f4e25f717e3760dcb048f9eea69578ad17d90c7f049f4b001bf84109a63"
Jul 17 20:30:02.234472 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:02.234449    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp_odh-ai-gateway-infra(0fc5cb1a-007d-43c5-97ab-282c74f4fd71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71"
Jul 17 20:30:02.238835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.238818    2572 scope.go:117] "RemoveContainer" containerID="55ebe470436fcd974100de0d5471c0627b35a0ed04c1c2393c3e58699b89fa5d"
Jul 17 20:30:02.286775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.286745    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"]
Jul 17 20:30:02.292635 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:02.292616    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8pzxv"]
Jul 17 20:30:03.240579 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:03.240545    2572 scope.go:117] "RemoveContainer" containerID="e44f5f4e25f717e3760dcb048f9eea69578ad17d90c7f049f4b001bf84109a63"
Jul 17 20:30:03.240951 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:03.240800    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp_odh-ai-gateway-infra(0fc5cb1a-007d-43c5-97ab-282c74f4fd71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71"
Jul 17 20:30:03.368033 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:03.368010    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc"
Jul 17 20:30:03.853064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:03.853021    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" path="/var/lib/kubelet/pods/6468fa0f-a251-4424-a991-26df34baa2cf/volumes"
Jul 17 20:30:04.245631 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:04.245606    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc"
Jul 17 20:30:04.246010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:04.245627    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738670-nspnc" event={"ID":"8f320eb5-cac2-48a5-a8bc-3c829238e69c","Type":"ContainerDied","Data":"3e5ecd70ae0227a6f70a53538a68fc216a1191cf9d1aae7f2a1aa76f646faa57"}
Jul 17 20:30:04.246010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:04.245678    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e5ecd70ae0227a6f70a53538a68fc216a1191cf9d1aae7f2a1aa76f646faa57"
Jul 17 20:30:07.848670 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:07.848640    2572 scope.go:117] "RemoveContainer" containerID="c7ff305bb92f2b44001734cb17c895d935138d25e6d70640315408aed007fcd4"
Jul 17 20:30:08.264397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:08.264363    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" event={"ID":"76038b90-5283-4c95-8e4a-6d399fc1d3b1","Type":"ContainerStarted","Data":"3da2f5493d131fdc04b59ca816ec4324f5cb83bdb5955b9ca8e06dacf87c6a69"}
Jul 17 20:30:08.284169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:08.284120    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" podStartSLOduration=20.284105782 podStartE2EDuration="20.284105782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:30:08.282337082 +0000 UTC m=+2042.976313228" watchObservedRunningTime="2026-07-17 20:30:08.284105782 +0000 UTC m=+2042.978081945"
Jul 17 20:30:09.270327 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:09.270299    2572 generic.go:356] "Generic (PLEG): container finished" podID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerID="3da2f5493d131fdc04b59ca816ec4324f5cb83bdb5955b9ca8e06dacf87c6a69" exitCode=6
Jul 17 20:30:09.270730 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:09.270380    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" event={"ID":"76038b90-5283-4c95-8e4a-6d399fc1d3b1","Type":"ContainerDied","Data":"3da2f5493d131fdc04b59ca816ec4324f5cb83bdb5955b9ca8e06dacf87c6a69"}
Jul 17 20:30:09.270730 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:09.270447    2572 scope.go:117] "RemoveContainer" containerID="c7ff305bb92f2b44001734cb17c895d935138d25e6d70640315408aed007fcd4"
Jul 17 20:30:09.270867 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:09.270715    2572 scope.go:117] "RemoveContainer" containerID="3da2f5493d131fdc04b59ca816ec4324f5cb83bdb5955b9ca8e06dacf87c6a69"
Jul 17 20:30:09.270963 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:09.270944    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz_odh-ai-gateway-infra(76038b90-5283-4c95-8e4a-6d399fc1d3b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1"
Jul 17 20:30:09.304535 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:09.304496    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"]
Jul 17 20:30:10.398373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:10.398348    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"
Jul 17 20:30:10.442249 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:10.442217    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76038b90-5283-4c95-8e4a-6d399fc1d3b1-maas-api-service-ca\") pod \"76038b90-5283-4c95-8e4a-6d399fc1d3b1\" (UID: \"76038b90-5283-4c95-8e4a-6d399fc1d3b1\") "
Jul 17 20:30:10.442624 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:10.442596    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76038b90-5283-4c95-8e4a-6d399fc1d3b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76038b90-5283-4c95-8e4a-6d399fc1d3b1" (UID: "76038b90-5283-4c95-8e4a-6d399fc1d3b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:30:10.543102 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:10.542983    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76038b90-5283-4c95-8e4a-6d399fc1d3b1-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:30:11.281489 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:11.281450    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz" event={"ID":"76038b90-5283-4c95-8e4a-6d399fc1d3b1","Type":"ContainerDied","Data":"5fdf12f5ec1b3122674f57d421cc5ad6c364b8340601911811f88258c2ee6f70"}
Jul 17 20:30:11.281489 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:11.281471    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"
Jul 17 20:30:11.281932 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:11.281556    2572 scope.go:117] "RemoveContainer" containerID="3da2f5493d131fdc04b59ca816ec4324f5cb83bdb5955b9ca8e06dacf87c6a69"
Jul 17 20:30:11.307213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:11.307186    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"]
Jul 17 20:30:11.325755 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:11.325730    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nb4qz"]
Jul 17 20:30:11.853318 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:11.853283    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" path="/var/lib/kubelet/pods/76038b90-5283-4c95-8e4a-6d399fc1d3b1/volumes"
Jul 17 20:30:16.848523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:16.848493    2572 scope.go:117] "RemoveContainer" containerID="e44f5f4e25f717e3760dcb048f9eea69578ad17d90c7f049f4b001bf84109a63"
Jul 17 20:30:17.313098 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:17.313040    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" event={"ID":"0fc5cb1a-007d-43c5-97ab-282c74f4fd71","Type":"ContainerStarted","Data":"0da63db96fe001ce4b7c52c12b5873c09ee3c8e24496c8e536f09f4a58fccf3b"}
Jul 17 20:30:17.329960 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:17.329912    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" podStartSLOduration=17.329898688 podStartE2EDuration="17.329898688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:30:17.329619473 +0000 UTC m=+2052.023595622" watchObservedRunningTime="2026-07-17 20:30:17.329898688 +0000 UTC m=+2052.023874812"
Jul 17 20:30:18.319023 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:18.318989    2572 generic.go:356] "Generic (PLEG): container finished" podID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerID="0da63db96fe001ce4b7c52c12b5873c09ee3c8e24496c8e536f09f4a58fccf3b" exitCode=7
Jul 17 20:30:18.319492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:18.319079    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" event={"ID":"0fc5cb1a-007d-43c5-97ab-282c74f4fd71","Type":"ContainerDied","Data":"0da63db96fe001ce4b7c52c12b5873c09ee3c8e24496c8e536f09f4a58fccf3b"}
Jul 17 20:30:18.319492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:18.319156    2572 scope.go:117] "RemoveContainer" containerID="e44f5f4e25f717e3760dcb048f9eea69578ad17d90c7f049f4b001bf84109a63"
Jul 17 20:30:18.319492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:18.319421    2572 scope.go:117] "RemoveContainer" containerID="0da63db96fe001ce4b7c52c12b5873c09ee3c8e24496c8e536f09f4a58fccf3b"
Jul 17 20:30:18.319655 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:18.319632    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp_odh-ai-gateway-infra(0fc5cb1a-007d-43c5-97ab-282c74f4fd71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71"
Jul 17 20:30:18.339481 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:18.339454    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"]
Jul 17 20:30:19.456316 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:19.456295    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"
Jul 17 20:30:20.330841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:20.330798    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp" event={"ID":"0fc5cb1a-007d-43c5-97ab-282c74f4fd71","Type":"ContainerDied","Data":"ca5b6826f05106677b8143023748ccfad569082d390d40b76e909ff8077dcc21"}
Jul 17 20:30:20.330841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:20.330829    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"
Jul 17 20:30:20.331069 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:20.330908    2572 scope.go:117] "RemoveContainer" containerID="0da63db96fe001ce4b7c52c12b5873c09ee3c8e24496c8e536f09f4a58fccf3b"
Jul 17 20:30:20.352787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:20.352757    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"]
Jul 17 20:30:20.356646 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:20.356624    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-9a08062a-29738670-c8gzp"]
Jul 17 20:30:21.853771 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:21.853735    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" path="/var/lib/kubelet/pods/0fc5cb1a-007d-43c5-97ab-282c74f4fd71/volumes"
Jul 17 20:30:36.999887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:36.999814    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"]
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000136    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000147    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000159    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000164    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000176    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f320eb5-cac2-48a5-a8bc-3c829238e69c" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000181    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f320eb5-cac2-48a5-a8bc-3c829238e69c" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000193    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000201    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000209    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000215    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000222    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000227    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000233    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000238    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000244    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000249    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000306    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f320eb5-cac2-48a5-a8bc-3c829238e69c" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000315    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000321    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000327    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000332    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000337    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:37.000373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.000342    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:37.004878 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.004850    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"
Jul 17 20:30:37.011101 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.011077    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:30:37.012095 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.011857    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:30:37.021652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.021634    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"]
Jul 17 20:30:37.058377 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.058346    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75dfa31-1bd6-4a78-8a64-19a9216fffc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc\" (UID: \"e75dfa31-1bd6-4a78-8a64-19a9216fffc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"
Jul 17 20:30:37.158916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.158884    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75dfa31-1bd6-4a78-8a64-19a9216fffc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc\" (UID: \"e75dfa31-1bd6-4a78-8a64-19a9216fffc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"
Jul 17 20:30:37.159567 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.159545    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75dfa31-1bd6-4a78-8a64-19a9216fffc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc\" (UID: \"e75dfa31-1bd6-4a78-8a64-19a9216fffc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"
Jul 17 20:30:37.315381 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.315296    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"
Jul 17 20:30:37.661971 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:37.661761    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"]
Jul 17 20:30:37.664577 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:30:37.664547    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75dfa31_1bd6_4a78_8a64_19a9216fffc1.slice/crio-1ee63a81f17b296ebe372163677e4213643b9a76232864f0fce7cb49e5c116e1 WatchSource:0}: Error finding container 1ee63a81f17b296ebe372163677e4213643b9a76232864f0fce7cb49e5c116e1: Status 404 returned error can't find the container with id 1ee63a81f17b296ebe372163677e4213643b9a76232864f0fce7cb49e5c116e1
Jul 17 20:30:38.410862 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:38.410830    2572 generic.go:356] "Generic (PLEG): container finished" podID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerID="6bd51529c904649c1d36c477c0bdeff94d76d6bfb344e0613790dadd98b975ac" exitCode=7
Jul 17 20:30:38.411265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:38.410872    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" event={"ID":"e75dfa31-1bd6-4a78-8a64-19a9216fffc1","Type":"ContainerDied","Data":"6bd51529c904649c1d36c477c0bdeff94d76d6bfb344e0613790dadd98b975ac"}
Jul 17 20:30:38.411265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:38.410931    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" event={"ID":"e75dfa31-1bd6-4a78-8a64-19a9216fffc1","Type":"ContainerStarted","Data":"1ee63a81f17b296ebe372163677e4213643b9a76232864f0fce7cb49e5c116e1"}
Jul 17 20:30:38.411265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:38.411113    2572 scope.go:117] "RemoveContainer" containerID="6bd51529c904649c1d36c477c0bdeff94d76d6bfb344e0613790dadd98b975ac"
Jul 17 20:30:39.417533 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:39.417449    2572 generic.go:356] "Generic (PLEG): container finished" podID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerID="6841e0bd0a892713771bb23fd0ec106a7215ef6f22552654da0d543aa712923b" exitCode=7
Jul 17 20:30:39.417533 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:39.417491    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" event={"ID":"e75dfa31-1bd6-4a78-8a64-19a9216fffc1","Type":"ContainerDied","Data":"6841e0bd0a892713771bb23fd0ec106a7215ef6f22552654da0d543aa712923b"}
Jul 17 20:30:39.417943 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:39.417562    2572 scope.go:117] "RemoveContainer" containerID="6bd51529c904649c1d36c477c0bdeff94d76d6bfb344e0613790dadd98b975ac"
Jul 17 20:30:39.417943 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:39.417689    2572 scope.go:117] "RemoveContainer" containerID="6841e0bd0a892713771bb23fd0ec106a7215ef6f22552654da0d543aa712923b"
Jul 17 20:30:39.417943 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:39.417902    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc_odh-ai-gateway-infra(e75dfa31-1bd6-4a78-8a64-19a9216fffc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1"
Jul 17 20:30:40.424160 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.424129    2572 scope.go:117] "RemoveContainer" containerID="6841e0bd0a892713771bb23fd0ec106a7215ef6f22552654da0d543aa712923b"
Jul 17 20:30:40.424613 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:40.424356    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc_odh-ai-gateway-infra(e75dfa31-1bd6-4a78-8a64-19a9216fffc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1"
Jul 17 20:30:40.715751 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.715722    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"]
Jul 17 20:30:40.716113 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.716030    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:40.716113 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.716089    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:40.716113 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.716107    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:40.716113 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.716115    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:40.716264 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.716181    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76038b90-5283-4c95-8e4a-6d399fc1d3b1" containerName="revoke-keys"
Jul 17 20:30:40.716264 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.716191    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6468fa0f-a251-4424-a991-26df34baa2cf" containerName="revoke-keys"
Jul 17 20:30:40.716264 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.716198    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc5cb1a-007d-43c5-97ab-282c74f4fd71" containerName="cleanup"
Jul 17 20:30:40.718889 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.718864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"
Jul 17 20:30:40.728790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.728764    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"]
Jul 17 20:30:40.788540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.788510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-hkh4j\" (UID: \"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"
Jul 17 20:30:40.889027 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.888997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-hkh4j\" (UID: \"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"
Jul 17 20:30:40.889611 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:40.889592    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-hkh4j\" (UID: \"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"
Jul 17 20:30:41.029217 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:41.029137    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"
Jul 17 20:30:41.156237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:41.156214    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"]
Jul 17 20:30:41.158307 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:30:41.158275    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e9dc15_2047_4c80_8bce_6f54c2ff6b2e.slice/crio-161a48e56935f5f7b7000a4dd8163a5f32a27a557c8d800b9ccb364bc0df2f04 WatchSource:0}: Error finding container 161a48e56935f5f7b7000a4dd8163a5f32a27a557c8d800b9ccb364bc0df2f04: Status 404 returned error can't find the container with id 161a48e56935f5f7b7000a4dd8163a5f32a27a557c8d800b9ccb364bc0df2f04
Jul 17 20:30:41.430000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:41.429969    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" event={"ID":"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e","Type":"ContainerStarted","Data":"161a48e56935f5f7b7000a4dd8163a5f32a27a557c8d800b9ccb364bc0df2f04"}
Jul 17 20:30:42.436263 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:42.436227    2572 generic.go:356] "Generic (PLEG): container finished" podID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerID="c9fba9ced39a1426279f93d3af198bdebdc37df03609efc38c4616c436a7ee25" exitCode=6
Jul 17 20:30:42.436676 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:42.436324    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" event={"ID":"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e","Type":"ContainerDied","Data":"c9fba9ced39a1426279f93d3af198bdebdc37df03609efc38c4616c436a7ee25"}
Jul 17 20:30:42.436676 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:42.436476    2572 scope.go:117] "RemoveContainer" containerID="c9fba9ced39a1426279f93d3af198bdebdc37df03609efc38c4616c436a7ee25"
Jul 17 20:30:43.442713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:43.442629    2572 generic.go:356] "Generic (PLEG): container finished" podID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerID="56d140f92fa2b508264a466f4f1a2d9fcfa379bd1b18d07d9910196ae733a134" exitCode=6
Jul 17 20:30:43.443114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:43.442715    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" event={"ID":"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e","Type":"ContainerDied","Data":"56d140f92fa2b508264a466f4f1a2d9fcfa379bd1b18d07d9910196ae733a134"}
Jul 17 20:30:43.443114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:43.442767    2572 scope.go:117] "RemoveContainer" containerID="c9fba9ced39a1426279f93d3af198bdebdc37df03609efc38c4616c436a7ee25"
Jul 17 20:30:43.443114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:43.442887    2572 scope.go:117] "RemoveContainer" containerID="56d140f92fa2b508264a466f4f1a2d9fcfa379bd1b18d07d9910196ae733a134"
Jul 17 20:30:43.443232 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:43.443126    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-hkh4j_odh-ai-gateway-infra(f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e"
Jul 17 20:30:44.449641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:44.449610    2572 scope.go:117] "RemoveContainer" containerID="56d140f92fa2b508264a466f4f1a2d9fcfa379bd1b18d07d9910196ae733a134"
Jul 17 20:30:44.450028 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:44.449768    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-hkh4j_odh-ai-gateway-infra(f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e"
Jul 17 20:30:48.784157 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:48.784121    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"]
Jul 17 20:30:48.788377 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:48.788359    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"
Jul 17 20:30:48.801541 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:48.801511    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"]
Jul 17 20:30:48.952553 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:48.952512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccb37dd-81f2-4f20-98bf-9648a3b01ccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh\" (UID: \"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"
Jul 17 20:30:49.053372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:49.053284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccb37dd-81f2-4f20-98bf-9648a3b01ccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh\" (UID: \"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"
Jul 17 20:30:49.053913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:49.053893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccb37dd-81f2-4f20-98bf-9648a3b01ccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh\" (UID: \"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"
Jul 17 20:30:49.098407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:49.098376    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"
Jul 17 20:30:49.238127 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:49.238104    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"]
Jul 17 20:30:49.240662 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:30:49.240637    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccb37dd_81f2_4f20_98bf_9648a3b01ccf.slice/crio-db76eecc9fc30abb9c8af10e3341c4ec33403e4121af5750a16f70260cf3fbe3 WatchSource:0}: Error finding container db76eecc9fc30abb9c8af10e3341c4ec33403e4121af5750a16f70260cf3fbe3: Status 404 returned error can't find the container with id db76eecc9fc30abb9c8af10e3341c4ec33403e4121af5750a16f70260cf3fbe3
Jul 17 20:30:49.476822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:49.476793    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" event={"ID":"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf","Type":"ContainerStarted","Data":"db76eecc9fc30abb9c8af10e3341c4ec33403e4121af5750a16f70260cf3fbe3"}
Jul 17 20:30:50.483388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:50.483358    2572 generic.go:356] "Generic (PLEG): container finished" podID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerID="afddf5c64ebeb6bfa7aad6e38f53de4a8faaefc85fe20c2fcff5d975d185be87" exitCode=6
Jul 17 20:30:50.483787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:50.483445    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" event={"ID":"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf","Type":"ContainerDied","Data":"afddf5c64ebeb6bfa7aad6e38f53de4a8faaefc85fe20c2fcff5d975d185be87"}
Jul 17 20:30:50.483787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:50.483589    2572 scope.go:117] "RemoveContainer" containerID="afddf5c64ebeb6bfa7aad6e38f53de4a8faaefc85fe20c2fcff5d975d185be87"
Jul 17 20:30:51.489124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:51.489094    2572 generic.go:356] "Generic (PLEG): container finished" podID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerID="c27916079c7653f91f647d3a755aafd8ce87ef65abaa6df150ab58a5ea61ffc1" exitCode=6
Jul 17 20:30:51.489558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:51.489184    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" event={"ID":"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf","Type":"ContainerDied","Data":"c27916079c7653f91f647d3a755aafd8ce87ef65abaa6df150ab58a5ea61ffc1"}
Jul 17 20:30:51.489558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:51.489254    2572 scope.go:117] "RemoveContainer" containerID="afddf5c64ebeb6bfa7aad6e38f53de4a8faaefc85fe20c2fcff5d975d185be87"
Jul 17 20:30:51.489558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:51.489380    2572 scope.go:117] "RemoveContainer" containerID="c27916079c7653f91f647d3a755aafd8ce87ef65abaa6df150ab58a5ea61ffc1"
Jul 17 20:30:51.489695 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:51.489570    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh_odh-ai-gateway-infra(3ccb37dd-81f2-4f20-98bf-9648a3b01ccf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf"
Jul 17 20:30:52.495599 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:52.495574    2572 scope.go:117] "RemoveContainer" containerID="c27916079c7653f91f647d3a755aafd8ce87ef65abaa6df150ab58a5ea61ffc1"
Jul 17 20:30:52.495968 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:52.495729    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh_odh-ai-gateway-infra(3ccb37dd-81f2-4f20-98bf-9648a3b01ccf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf"
Jul 17 20:30:55.851399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:55.851367    2572 scope.go:117] "RemoveContainer" containerID="6841e0bd0a892713771bb23fd0ec106a7215ef6f22552654da0d543aa712923b"
Jul 17 20:30:56.514350 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:56.514318    2572 generic.go:356] "Generic (PLEG): container finished" podID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerID="af031a5e20475ad20b41c0cc5fcfe018b970af6a9e5ba6d03ecdbc74ee07073c" exitCode=7
Jul 17 20:30:56.514509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:56.514386    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" event={"ID":"e75dfa31-1bd6-4a78-8a64-19a9216fffc1","Type":"ContainerDied","Data":"af031a5e20475ad20b41c0cc5fcfe018b970af6a9e5ba6d03ecdbc74ee07073c"}
Jul 17 20:30:56.514509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:56.514441    2572 scope.go:117] "RemoveContainer" containerID="6841e0bd0a892713771bb23fd0ec106a7215ef6f22552654da0d543aa712923b"
Jul 17 20:30:56.514733 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:56.514713    2572 scope.go:117] "RemoveContainer" containerID="af031a5e20475ad20b41c0cc5fcfe018b970af6a9e5ba6d03ecdbc74ee07073c"
Jul 17 20:30:56.514912 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:56.514896    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc_odh-ai-gateway-infra(e75dfa31-1bd6-4a78-8a64-19a9216fffc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1"
Jul 17 20:30:57.552952 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:57.552919    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"]
Jul 17 20:30:57.684018 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:57.683991    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"
Jul 17 20:30:57.827000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:57.826902    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75dfa31-1bd6-4a78-8a64-19a9216fffc1-maas-api-service-ca\") pod \"e75dfa31-1bd6-4a78-8a64-19a9216fffc1\" (UID: \"e75dfa31-1bd6-4a78-8a64-19a9216fffc1\") "
Jul 17 20:30:57.827327 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:57.827299    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75dfa31-1bd6-4a78-8a64-19a9216fffc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e75dfa31-1bd6-4a78-8a64-19a9216fffc1" (UID: "e75dfa31-1bd6-4a78-8a64-19a9216fffc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:30:57.848448 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:57.848414    2572 scope.go:117] "RemoveContainer" containerID="56d140f92fa2b508264a466f4f1a2d9fcfa379bd1b18d07d9910196ae733a134"
Jul 17 20:30:57.927866 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:57.927836    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75dfa31-1bd6-4a78-8a64-19a9216fffc1-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:30:58.525958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.525925    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"
Jul 17 20:30:58.526189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.525962    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc" event={"ID":"e75dfa31-1bd6-4a78-8a64-19a9216fffc1","Type":"ContainerDied","Data":"1ee63a81f17b296ebe372163677e4213643b9a76232864f0fce7cb49e5c116e1"}
Jul 17 20:30:58.526189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.526034    2572 scope.go:117] "RemoveContainer" containerID="af031a5e20475ad20b41c0cc5fcfe018b970af6a9e5ba6d03ecdbc74ee07073c"
Jul 17 20:30:58.527883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.527857    2572 generic.go:356] "Generic (PLEG): container finished" podID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerID="0a36ab9480f45f9f0d1de712f3827a7cbae9d375164bca080b060193be47d9dc" exitCode=6
Jul 17 20:30:58.528010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.527894    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" event={"ID":"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e","Type":"ContainerDied","Data":"0a36ab9480f45f9f0d1de712f3827a7cbae9d375164bca080b060193be47d9dc"}
Jul 17 20:30:58.528359 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.528343    2572 scope.go:117] "RemoveContainer" containerID="0a36ab9480f45f9f0d1de712f3827a7cbae9d375164bca080b060193be47d9dc"
Jul 17 20:30:58.528586 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:30:58.528568    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-hkh4j_odh-ai-gateway-infra(f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e"
Jul 17 20:30:58.534009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.533994    2572 scope.go:117] "RemoveContainer" containerID="56d140f92fa2b508264a466f4f1a2d9fcfa379bd1b18d07d9910196ae733a134"
Jul 17 20:30:58.550784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.550759    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"]
Jul 17 20:30:58.560961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:58.560940    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kmzmc"]
Jul 17 20:30:59.616610 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:59.616576    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"]
Jul 17 20:30:59.751551 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:59.751523    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"
Jul 17 20:30:59.844162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:59.844132    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e-maas-api-service-ca\") pod \"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e\" (UID: \"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e\") "
Jul 17 20:30:59.844494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:59.844472    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" (UID: "f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:30:59.853262 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:59.853235    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" path="/var/lib/kubelet/pods/e75dfa31-1bd6-4a78-8a64-19a9216fffc1/volumes"
Jul 17 20:30:59.945446 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:30:59.945359    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:31:00.542535 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:00.542499    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"
Jul 17 20:31:00.542704 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:00.542496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j" event={"ID":"f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e","Type":"ContainerDied","Data":"161a48e56935f5f7b7000a4dd8163a5f32a27a557c8d800b9ccb364bc0df2f04"}
Jul 17 20:31:00.542704 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:00.542641    2572 scope.go:117] "RemoveContainer" containerID="0a36ab9480f45f9f0d1de712f3827a7cbae9d375164bca080b060193be47d9dc"
Jul 17 20:31:00.566975 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:00.566943    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"]
Jul 17 20:31:00.573348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:00.573328    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hkh4j"]
Jul 17 20:31:01.852895 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:01.852865    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" path="/var/lib/kubelet/pods/f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e/volumes"
Jul 17 20:31:04.848286 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:04.848257    2572 scope.go:117] "RemoveContainer" containerID="c27916079c7653f91f647d3a755aafd8ce87ef65abaa6df150ab58a5ea61ffc1"
Jul 17 20:31:05.565214 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:05.565184    2572 generic.go:356] "Generic (PLEG): container finished" podID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerID="c161c0ca2010deece7d39056476df5fe364f73197ebce95a0fca19e8a2e79248" exitCode=6
Jul 17 20:31:05.565384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:05.565261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" event={"ID":"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf","Type":"ContainerDied","Data":"c161c0ca2010deece7d39056476df5fe364f73197ebce95a0fca19e8a2e79248"}
Jul 17 20:31:05.565384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:05.565319    2572 scope.go:117] "RemoveContainer" containerID="c27916079c7653f91f647d3a755aafd8ce87ef65abaa6df150ab58a5ea61ffc1"
Jul 17 20:31:05.565635 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:05.565621    2572 scope.go:117] "RemoveContainer" containerID="c161c0ca2010deece7d39056476df5fe364f73197ebce95a0fca19e8a2e79248"
Jul 17 20:31:05.565849 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:05.565826    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh_odh-ai-gateway-infra(3ccb37dd-81f2-4f20-98bf-9648a3b01ccf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf"
Jul 17 20:31:06.603400 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:06.603364    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"]
Jul 17 20:31:06.735135 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:06.735108    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"
Jul 17 20:31:06.799416 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:06.799387    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccb37dd-81f2-4f20-98bf-9648a3b01ccf-maas-api-service-ca\") pod \"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf\" (UID: \"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf\") "
Jul 17 20:31:06.799721 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:06.799698    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ccb37dd-81f2-4f20-98bf-9648a3b01ccf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" (UID: "3ccb37dd-81f2-4f20-98bf-9648a3b01ccf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:31:06.899949 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:06.899870    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccb37dd-81f2-4f20-98bf-9648a3b01ccf-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:31:07.576773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:07.576742    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"
Jul 17 20:31:07.576963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:07.576779    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh" event={"ID":"3ccb37dd-81f2-4f20-98bf-9648a3b01ccf","Type":"ContainerDied","Data":"db76eecc9fc30abb9c8af10e3341c4ec33403e4121af5750a16f70260cf3fbe3"}
Jul 17 20:31:07.576963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:07.576847    2572 scope.go:117] "RemoveContainer" containerID="c161c0ca2010deece7d39056476df5fe364f73197ebce95a0fca19e8a2e79248"
Jul 17 20:31:07.607458 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:07.607428    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"]
Jul 17 20:31:07.617559 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:07.617536    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-gb4bh"]
Jul 17 20:31:07.855851 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:07.855766    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" path="/var/lib/kubelet/pods/3ccb37dd-81f2-4f20-98bf-9648a3b01ccf/volumes"
Jul 17 20:31:30.792200 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792171    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"]
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792457    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792467    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792480    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792485    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792491    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792497    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792508    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792513    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792520    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792525    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792532    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792538    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792543    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792548    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792596    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792603    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792610    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792615    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792621    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:30.792665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.792627    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:30.795661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.795644    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"
Jul 17 20:31:30.798143 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.798123    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:31:30.798143 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.798139    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:31:30.807262 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.807238    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"]
Jul 17 20:31:30.874951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.874916    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fpjgf\" (UID: \"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"
Jul 17 20:31:30.975298 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.975260    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fpjgf\" (UID: \"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"
Jul 17 20:31:30.975898 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:30.975878    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fpjgf\" (UID: \"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"
Jul 17 20:31:31.106277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:31.106191    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"
Jul 17 20:31:31.245466 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:31.245437    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"]
Jul 17 20:31:31.248090 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:31:31.247916    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77060b2b_1f11_4f8e_b5b9_8a5cdcdc53d9.slice/crio-7cabb78d3d1b52170f74ed3198ea346c22bca5a45f6f64cf6adccc13cb26774e WatchSource:0}: Error finding container 7cabb78d3d1b52170f74ed3198ea346c22bca5a45f6f64cf6adccc13cb26774e: Status 404 returned error can't find the container with id 7cabb78d3d1b52170f74ed3198ea346c22bca5a45f6f64cf6adccc13cb26774e
Jul 17 20:31:31.683314 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:31.683275    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" event={"ID":"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9","Type":"ContainerStarted","Data":"d3d15988d0b6a6f13eff9d700fbdb31bb68058de26c863467f1045dbc1da9267"}
Jul 17 20:31:31.683500 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:31.683362    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" event={"ID":"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9","Type":"ContainerStarted","Data":"7cabb78d3d1b52170f74ed3198ea346c22bca5a45f6f64cf6adccc13cb26774e"}
Jul 17 20:31:31.703487 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:31.703402    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" podStartSLOduration=1.703389013 podStartE2EDuration="1.703389013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:31:31.701874195 +0000 UTC m=+2126.395850340" watchObservedRunningTime="2026-07-17 20:31:31.703389013 +0000 UTC m=+2126.397365160"
Jul 17 20:31:32.689388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:32.689355    2572 generic.go:356] "Generic (PLEG): container finished" podID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerID="d3d15988d0b6a6f13eff9d700fbdb31bb68058de26c863467f1045dbc1da9267" exitCode=6
Jul 17 20:31:32.689775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:32.689423    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" event={"ID":"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9","Type":"ContainerDied","Data":"d3d15988d0b6a6f13eff9d700fbdb31bb68058de26c863467f1045dbc1da9267"}
Jul 17 20:31:32.689775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:32.689613    2572 scope.go:117] "RemoveContainer" containerID="d3d15988d0b6a6f13eff9d700fbdb31bb68058de26c863467f1045dbc1da9267"
Jul 17 20:31:33.696378 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:33.696288    2572 generic.go:356] "Generic (PLEG): container finished" podID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerID="b7756d182d51b87d9fb47a3987f0cd44c5b63ebc0afe55b9c99a33097db44f50" exitCode=6
Jul 17 20:31:33.696378 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:33.696354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" event={"ID":"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9","Type":"ContainerDied","Data":"b7756d182d51b87d9fb47a3987f0cd44c5b63ebc0afe55b9c99a33097db44f50"}
Jul 17 20:31:33.696889 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:33.696403    2572 scope.go:117] "RemoveContainer" containerID="d3d15988d0b6a6f13eff9d700fbdb31bb68058de26c863467f1045dbc1da9267"
Jul 17 20:31:33.696889 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:33.696643    2572 scope.go:117] "RemoveContainer" containerID="b7756d182d51b87d9fb47a3987f0cd44c5b63ebc0afe55b9c99a33097db44f50"
Jul 17 20:31:33.696889 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:33.696858    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fpjgf_odh-ai-gateway-infra(77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9"
Jul 17 20:31:34.701987 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:34.701955    2572 scope.go:117] "RemoveContainer" containerID="b7756d182d51b87d9fb47a3987f0cd44c5b63ebc0afe55b9c99a33097db44f50"
Jul 17 20:31:34.702402 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:34.702174    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fpjgf_odh-ai-gateway-infra(77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9"
Jul 17 20:31:37.002383 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002350    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"]
Jul 17 20:31:37.002773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002662    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:37.002773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002674    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:37.002773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002689    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:37.002773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002694    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:37.002773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002746    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e9dc15-2047-4c80-8bce-6f54c2ff6b2e" containerName="revoke-keys"
Jul 17 20:31:37.002773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002754    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccb37dd-81f2-4f20-98bf-9648a3b01ccf" containerName="revoke-keys"
Jul 17 20:31:37.002773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.002760    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e75dfa31-1bd6-4a78-8a64-19a9216fffc1" containerName="revoke-keys"
Jul 17 20:31:37.006764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.006748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"
Jul 17 20:31:37.016039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.016016    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"]
Jul 17 20:31:37.126944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.126911    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f0dcd6-e58e-4ca0-b55c-5631bec974a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5\" (UID: \"59f0dcd6-e58e-4ca0-b55c-5631bec974a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"
Jul 17 20:31:37.227930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.227895    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f0dcd6-e58e-4ca0-b55c-5631bec974a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5\" (UID: \"59f0dcd6-e58e-4ca0-b55c-5631bec974a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"
Jul 17 20:31:37.228564 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.228541    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f0dcd6-e58e-4ca0-b55c-5631bec974a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5\" (UID: \"59f0dcd6-e58e-4ca0-b55c-5631bec974a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"
Jul 17 20:31:37.316801 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.316718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"
Jul 17 20:31:37.441696 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.441672    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"]
Jul 17 20:31:37.443994 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:31:37.443970    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f0dcd6_e58e_4ca0_b55c_5631bec974a5.slice/crio-3a58e91dbddcd57efe60d66877bdb071ef318316718a04087fb16e512b6a95ce WatchSource:0}: Error finding container 3a58e91dbddcd57efe60d66877bdb071ef318316718a04087fb16e512b6a95ce: Status 404 returned error can't find the container with id 3a58e91dbddcd57efe60d66877bdb071ef318316718a04087fb16e512b6a95ce
Jul 17 20:31:37.715498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:37.715462    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" event={"ID":"59f0dcd6-e58e-4ca0-b55c-5631bec974a5","Type":"ContainerStarted","Data":"3a58e91dbddcd57efe60d66877bdb071ef318316718a04087fb16e512b6a95ce"}
Jul 17 20:31:38.721538 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:38.721468    2572 generic.go:356] "Generic (PLEG): container finished" podID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerID="c74cfeb1eb040b5214b4dcae75663831868858883d1de0d5b708bcebb2719b01" exitCode=7
Jul 17 20:31:38.721538 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:38.721518    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" event={"ID":"59f0dcd6-e58e-4ca0-b55c-5631bec974a5","Type":"ContainerDied","Data":"c74cfeb1eb040b5214b4dcae75663831868858883d1de0d5b708bcebb2719b01"}
Jul 17 20:31:38.721998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:38.721739    2572 scope.go:117] "RemoveContainer" containerID="c74cfeb1eb040b5214b4dcae75663831868858883d1de0d5b708bcebb2719b01"
Jul 17 20:31:38.839442 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:38.839410    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"]
Jul 17 20:31:38.843631 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:38.843613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"
Jul 17 20:31:38.860256 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:38.860232    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"]
Jul 17 20:31:38.941983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:38.941957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c166f982-9ef9-4688-abfa-61b33c770a26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw\" (UID: \"c166f982-9ef9-4688-abfa-61b33c770a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"
Jul 17 20:31:39.043232 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.043205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c166f982-9ef9-4688-abfa-61b33c770a26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw\" (UID: \"c166f982-9ef9-4688-abfa-61b33c770a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"
Jul 17 20:31:39.043804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.043785    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c166f982-9ef9-4688-abfa-61b33c770a26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw\" (UID: \"c166f982-9ef9-4688-abfa-61b33c770a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"
Jul 17 20:31:39.152582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.152546    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"
Jul 17 20:31:39.284764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.284740    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"]
Jul 17 20:31:39.286918 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:31:39.286881    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc166f982_9ef9_4688_abfa_61b33c770a26.slice/crio-7fc67b00b38220a3356df1a44f90c0ec35baed30c57e36861364819b9edbecfe WatchSource:0}: Error finding container 7fc67b00b38220a3356df1a44f90c0ec35baed30c57e36861364819b9edbecfe: Status 404 returned error can't find the container with id 7fc67b00b38220a3356df1a44f90c0ec35baed30c57e36861364819b9edbecfe
Jul 17 20:31:39.728075 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.728029    2572 generic.go:356] "Generic (PLEG): container finished" podID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerID="5d13328a709d5749f454322a7e4ea284fa8f61ec80314ddfae67709805b62b5f" exitCode=7
Jul 17 20:31:39.728559 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.728119    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" event={"ID":"59f0dcd6-e58e-4ca0-b55c-5631bec974a5","Type":"ContainerDied","Data":"5d13328a709d5749f454322a7e4ea284fa8f61ec80314ddfae67709805b62b5f"}
Jul 17 20:31:39.728559 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.728180    2572 scope.go:117] "RemoveContainer" containerID="c74cfeb1eb040b5214b4dcae75663831868858883d1de0d5b708bcebb2719b01"
Jul 17 20:31:39.728559 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.728363    2572 scope.go:117] "RemoveContainer" containerID="5d13328a709d5749f454322a7e4ea284fa8f61ec80314ddfae67709805b62b5f"
Jul 17 20:31:39.728739 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:39.728578    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5_odh-ai-gateway-infra(59f0dcd6-e58e-4ca0-b55c-5631bec974a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5"
Jul 17 20:31:39.729696 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.729678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" event={"ID":"c166f982-9ef9-4688-abfa-61b33c770a26","Type":"ContainerStarted","Data":"982504abf5439ae08f13daf98df831062c7481d65f17ccaa62786e46d10e09af"}
Jul 17 20:31:39.729748 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.729713    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" event={"ID":"c166f982-9ef9-4688-abfa-61b33c770a26","Type":"ContainerStarted","Data":"7fc67b00b38220a3356df1a44f90c0ec35baed30c57e36861364819b9edbecfe"}
Jul 17 20:31:39.765637 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:39.765593    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" podStartSLOduration=1.765578492 podStartE2EDuration="1.765578492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:31:39.764814597 +0000 UTC m=+2134.458790743" watchObservedRunningTime="2026-07-17 20:31:39.765578492 +0000 UTC m=+2134.459554638"
Jul 17 20:31:40.734684 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:40.734652    2572 generic.go:356] "Generic (PLEG): container finished" podID="c166f982-9ef9-4688-abfa-61b33c770a26" containerID="982504abf5439ae08f13daf98df831062c7481d65f17ccaa62786e46d10e09af" exitCode=6
Jul 17 20:31:40.735146 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:40.734728    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" event={"ID":"c166f982-9ef9-4688-abfa-61b33c770a26","Type":"ContainerDied","Data":"982504abf5439ae08f13daf98df831062c7481d65f17ccaa62786e46d10e09af"}
Jul 17 20:31:40.735146 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:40.734892    2572 scope.go:117] "RemoveContainer" containerID="982504abf5439ae08f13daf98df831062c7481d65f17ccaa62786e46d10e09af"
Jul 17 20:31:40.736542 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:40.736526    2572 scope.go:117] "RemoveContainer" containerID="5d13328a709d5749f454322a7e4ea284fa8f61ec80314ddfae67709805b62b5f"
Jul 17 20:31:40.736674 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:40.736658    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5_odh-ai-gateway-infra(59f0dcd6-e58e-4ca0-b55c-5631bec974a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5"
Jul 17 20:31:41.742082 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:41.742024    2572 generic.go:356] "Generic (PLEG): container finished" podID="c166f982-9ef9-4688-abfa-61b33c770a26" containerID="87074ec3290f873a3546f062e28802e4cee70403e945d1442178257a3e4c956d" exitCode=6
Jul 17 20:31:41.742539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:41.742108    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" event={"ID":"c166f982-9ef9-4688-abfa-61b33c770a26","Type":"ContainerDied","Data":"87074ec3290f873a3546f062e28802e4cee70403e945d1442178257a3e4c956d"}
Jul 17 20:31:41.742539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:41.742195    2572 scope.go:117] "RemoveContainer" containerID="982504abf5439ae08f13daf98df831062c7481d65f17ccaa62786e46d10e09af"
Jul 17 20:31:41.742539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:41.742403    2572 scope.go:117] "RemoveContainer" containerID="87074ec3290f873a3546f062e28802e4cee70403e945d1442178257a3e4c956d"
Jul 17 20:31:41.742663 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:41.742609    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw_odh-ai-gateway-infra(c166f982-9ef9-4688-abfa-61b33c770a26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" podUID="c166f982-9ef9-4688-abfa-61b33c770a26"
Jul 17 20:31:42.747717 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:42.747690    2572 scope.go:117] "RemoveContainer" containerID="87074ec3290f873a3546f062e28802e4cee70403e945d1442178257a3e4c956d"
Jul 17 20:31:42.748138 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:42.747876    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw_odh-ai-gateway-infra(c166f982-9ef9-4688-abfa-61b33c770a26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" podUID="c166f982-9ef9-4688-abfa-61b33c770a26"
Jul 17 20:31:47.848347 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:47.848310    2572 scope.go:117] "RemoveContainer" containerID="b7756d182d51b87d9fb47a3987f0cd44c5b63ebc0afe55b9c99a33097db44f50"
Jul 17 20:31:48.774106 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:48.774075    2572 generic.go:356] "Generic (PLEG): container finished" podID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerID="1d806c7786f0cb546e4ba754bfee7725b2a38fdeb33154b3aa7cd08b442d4ebe" exitCode=6
Jul 17 20:31:48.774289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:48.774121    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" event={"ID":"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9","Type":"ContainerDied","Data":"1d806c7786f0cb546e4ba754bfee7725b2a38fdeb33154b3aa7cd08b442d4ebe"}
Jul 17 20:31:48.774289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:48.774178    2572 scope.go:117] "RemoveContainer" containerID="b7756d182d51b87d9fb47a3987f0cd44c5b63ebc0afe55b9c99a33097db44f50"
Jul 17 20:31:48.774453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:48.774430    2572 scope.go:117] "RemoveContainer" containerID="1d806c7786f0cb546e4ba754bfee7725b2a38fdeb33154b3aa7cd08b442d4ebe"
Jul 17 20:31:48.774669 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:48.774647    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fpjgf_odh-ai-gateway-infra(77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9"
Jul 17 20:31:49.820606 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:49.820566    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"]
Jul 17 20:31:49.953141 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:49.953109    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"
Jul 17 20:31:50.031681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.031653    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9-maas-api-service-ca\") pod \"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9\" (UID: \"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9\") "
Jul 17 20:31:50.031987 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.031967    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" (UID: "77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:31:50.132398 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.132320    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:31:50.784787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.784758    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"
Jul 17 20:31:50.784984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.784759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf" event={"ID":"77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9","Type":"ContainerDied","Data":"7cabb78d3d1b52170f74ed3198ea346c22bca5a45f6f64cf6adccc13cb26774e"}
Jul 17 20:31:50.784984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.784895    2572 scope.go:117] "RemoveContainer" containerID="1d806c7786f0cb546e4ba754bfee7725b2a38fdeb33154b3aa7cd08b442d4ebe"
Jul 17 20:31:50.805553 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.805524    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"]
Jul 17 20:31:50.809713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:50.809689    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fpjgf"]
Jul 17 20:31:51.848171 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:51.848141    2572 scope.go:117] "RemoveContainer" containerID="5d13328a709d5749f454322a7e4ea284fa8f61ec80314ddfae67709805b62b5f"
Jul 17 20:31:51.857548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:51.857520    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" path="/var/lib/kubelet/pods/77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9/volumes"
Jul 17 20:31:52.796335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:52.796297    2572 generic.go:356] "Generic (PLEG): container finished" podID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerID="21a799c63e1e2999d4251122b60fdebbd75a92fecf958ef9e7c68a0b96884a89" exitCode=7
Jul 17 20:31:52.796479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:52.796372    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" event={"ID":"59f0dcd6-e58e-4ca0-b55c-5631bec974a5","Type":"ContainerDied","Data":"21a799c63e1e2999d4251122b60fdebbd75a92fecf958ef9e7c68a0b96884a89"}
Jul 17 20:31:52.796479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:52.796431    2572 scope.go:117] "RemoveContainer" containerID="5d13328a709d5749f454322a7e4ea284fa8f61ec80314ddfae67709805b62b5f"
Jul 17 20:31:52.796736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:52.796713    2572 scope.go:117] "RemoveContainer" containerID="21a799c63e1e2999d4251122b60fdebbd75a92fecf958ef9e7c68a0b96884a89"
Jul 17 20:31:52.796915 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:52.796895    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5_odh-ai-gateway-infra(59f0dcd6-e58e-4ca0-b55c-5631bec974a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5"
Jul 17 20:31:53.824085 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:53.824038    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"]
Jul 17 20:31:53.957900 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:53.957877    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"
Jul 17 20:31:54.066072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.066027    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f0dcd6-e58e-4ca0-b55c-5631bec974a5-maas-api-service-ca\") pod \"59f0dcd6-e58e-4ca0-b55c-5631bec974a5\" (UID: \"59f0dcd6-e58e-4ca0-b55c-5631bec974a5\") "
Jul 17 20:31:54.066392 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.066373    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f0dcd6-e58e-4ca0-b55c-5631bec974a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59f0dcd6-e58e-4ca0-b55c-5631bec974a5" (UID: "59f0dcd6-e58e-4ca0-b55c-5631bec974a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:31:54.167529 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.167444    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f0dcd6-e58e-4ca0-b55c-5631bec974a5-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:31:54.809605 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.809576    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5" event={"ID":"59f0dcd6-e58e-4ca0-b55c-5631bec974a5","Type":"ContainerDied","Data":"3a58e91dbddcd57efe60d66877bdb071ef318316718a04087fb16e512b6a95ce"}
Jul 17 20:31:54.809605 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.809601    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"
Jul 17 20:31:54.809917 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.809640    2572 scope.go:117] "RemoveContainer" containerID="21a799c63e1e2999d4251122b60fdebbd75a92fecf958ef9e7c68a0b96884a89"
Jul 17 20:31:54.831160 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.831133    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"]
Jul 17 20:31:54.836622 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:54.836597    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-6c4f5"]
Jul 17 20:31:55.853552 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:55.853519    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" path="/var/lib/kubelet/pods/59f0dcd6-e58e-4ca0-b55c-5631bec974a5/volumes"
Jul 17 20:31:57.847936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:57.847862    2572 scope.go:117] "RemoveContainer" containerID="87074ec3290f873a3546f062e28802e4cee70403e945d1442178257a3e4c956d"
Jul 17 20:31:58.828855 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:58.828824    2572 generic.go:356] "Generic (PLEG): container finished" podID="c166f982-9ef9-4688-abfa-61b33c770a26" containerID="63c2ac58c0ceec4f34347993e3c720d2bf8babea36cd7528b9afbc754e655497" exitCode=6
Jul 17 20:31:58.829086 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:58.828900    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" event={"ID":"c166f982-9ef9-4688-abfa-61b33c770a26","Type":"ContainerDied","Data":"63c2ac58c0ceec4f34347993e3c720d2bf8babea36cd7528b9afbc754e655497"}
Jul 17 20:31:58.829086 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:58.829000    2572 scope.go:117] "RemoveContainer" containerID="87074ec3290f873a3546f062e28802e4cee70403e945d1442178257a3e4c956d"
Jul 17 20:31:58.829382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:58.829361    2572 scope.go:117] "RemoveContainer" containerID="63c2ac58c0ceec4f34347993e3c720d2bf8babea36cd7528b9afbc754e655497"
Jul 17 20:31:58.829576 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:31:58.829552    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw_odh-ai-gateway-infra(c166f982-9ef9-4688-abfa-61b33c770a26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" podUID="c166f982-9ef9-4688-abfa-61b33c770a26"
Jul 17 20:31:59.856479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:59.856450    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"]
Jul 17 20:31:59.983889 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:31:59.983863    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"
Jul 17 20:32:00.116109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.116004    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c166f982-9ef9-4688-abfa-61b33c770a26-maas-api-service-ca\") pod \"c166f982-9ef9-4688-abfa-61b33c770a26\" (UID: \"c166f982-9ef9-4688-abfa-61b33c770a26\") "
Jul 17 20:32:00.116384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.116363    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c166f982-9ef9-4688-abfa-61b33c770a26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c166f982-9ef9-4688-abfa-61b33c770a26" (UID: "c166f982-9ef9-4688-abfa-61b33c770a26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:32:00.217334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.217305    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c166f982-9ef9-4688-abfa-61b33c770a26-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:32:00.842322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.842297    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"
Jul 17 20:32:00.842537 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.842294    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw" event={"ID":"c166f982-9ef9-4688-abfa-61b33c770a26","Type":"ContainerDied","Data":"7fc67b00b38220a3356df1a44f90c0ec35baed30c57e36861364819b9edbecfe"}
Jul 17 20:32:00.842537 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.842450    2572 scope.go:117] "RemoveContainer" containerID="63c2ac58c0ceec4f34347993e3c720d2bf8babea36cd7528b9afbc754e655497"
Jul 17 20:32:00.866665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.866632    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"]
Jul 17 20:32:00.868492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:00.868471    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4zvbw"]
Jul 17 20:32:01.852295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:01.852261    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" path="/var/lib/kubelet/pods/c166f982-9ef9-4688-abfa-61b33c770a26/volumes"
Jul 17 20:32:27.053368 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053329    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"]
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053638    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053649    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053663    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053668    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053676    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053681    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053687    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053692    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053697    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053704    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053710    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053716    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053722    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053727    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053733    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053737    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053786    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053792    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053798    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053803    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f0dcd6-e58e-4ca0-b55c-5631bec974a5" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053808    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053814    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053820    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:27.053813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053825    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c166f982-9ef9-4688-abfa-61b33c770a26" containerName="revoke-keys"
Jul 17 20:32:27.054555 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.053831    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:27.056756 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.056739    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"
Jul 17 20:32:27.059250 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.059230    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:32:27.061361 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.061343    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:32:27.067808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.067782    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"]
Jul 17 20:32:27.123112 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.123042    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5772bde1-4c04-4a8c-85b6-d883e1b54231-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-grwx2\" (UID: \"5772bde1-4c04-4a8c-85b6-d883e1b54231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"
Jul 17 20:32:27.223574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.223539    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5772bde1-4c04-4a8c-85b6-d883e1b54231-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-grwx2\" (UID: \"5772bde1-4c04-4a8c-85b6-d883e1b54231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"
Jul 17 20:32:27.224180 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.224161    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5772bde1-4c04-4a8c-85b6-d883e1b54231-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-grwx2\" (UID: \"5772bde1-4c04-4a8c-85b6-d883e1b54231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"
Jul 17 20:32:27.366439 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.366357    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"
Jul 17 20:32:27.502603 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.502574    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"]
Jul 17 20:32:27.504501 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:32:27.504465    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5772bde1_4c04_4a8c_85b6_d883e1b54231.slice/crio-e328a243118b67f9c91adadda3780b601c7f2235e0763d60d4e518c416369c16 WatchSource:0}: Error finding container e328a243118b67f9c91adadda3780b601c7f2235e0763d60d4e518c416369c16: Status 404 returned error can't find the container with id e328a243118b67f9c91adadda3780b601c7f2235e0763d60d4e518c416369c16
Jul 17 20:32:27.956758 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.956723    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" event={"ID":"5772bde1-4c04-4a8c-85b6-d883e1b54231","Type":"ContainerStarted","Data":"34b844db7ed8c86843aacfb285da09874643f7e5e9ddc4064d06ad2782d35ae0"}
Jul 17 20:32:27.956919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.956773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" event={"ID":"5772bde1-4c04-4a8c-85b6-d883e1b54231","Type":"ContainerStarted","Data":"e328a243118b67f9c91adadda3780b601c7f2235e0763d60d4e518c416369c16"}
Jul 17 20:32:27.972793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:27.972749    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" podStartSLOduration=0.972733877 podStartE2EDuration="972.733877ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:32:27.972272887 +0000 UTC m=+2182.666249034" watchObservedRunningTime="2026-07-17 20:32:27.972733877 +0000 UTC m=+2182.666710021"
Jul 17 20:32:28.968759 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:28.968724    2572 generic.go:356] "Generic (PLEG): container finished" podID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerID="34b844db7ed8c86843aacfb285da09874643f7e5e9ddc4064d06ad2782d35ae0" exitCode=7
Jul 17 20:32:28.969186 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:28.968811    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" event={"ID":"5772bde1-4c04-4a8c-85b6-d883e1b54231","Type":"ContainerDied","Data":"34b844db7ed8c86843aacfb285da09874643f7e5e9ddc4064d06ad2782d35ae0"}
Jul 17 20:32:28.969186 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:28.968948    2572 scope.go:117] "RemoveContainer" containerID="34b844db7ed8c86843aacfb285da09874643f7e5e9ddc4064d06ad2782d35ae0"
Jul 17 20:32:29.974376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:29.974346    2572 generic.go:356] "Generic (PLEG): container finished" podID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerID="32cc5439b49a156379a91089d15d8e0092864d6d3614c7cfb05fe36f60d704ac" exitCode=7
Jul 17 20:32:29.974768 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:29.974430    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" event={"ID":"5772bde1-4c04-4a8c-85b6-d883e1b54231","Type":"ContainerDied","Data":"32cc5439b49a156379a91089d15d8e0092864d6d3614c7cfb05fe36f60d704ac"}
Jul 17 20:32:29.974768 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:29.974488    2572 scope.go:117] "RemoveContainer" containerID="34b844db7ed8c86843aacfb285da09874643f7e5e9ddc4064d06ad2782d35ae0"
Jul 17 20:32:29.974768 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:29.974724    2572 scope.go:117] "RemoveContainer" containerID="32cc5439b49a156379a91089d15d8e0092864d6d3614c7cfb05fe36f60d704ac"
Jul 17 20:32:29.974943 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:29.974920    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-grwx2_odh-ai-gateway-infra(5772bde1-4c04-4a8c-85b6-d883e1b54231)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231"
Jul 17 20:32:30.838515 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:30.838474    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"]
Jul 17 20:32:30.838814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:30.838802    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:30.838859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:30.838816    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="77060b2b-1f11-4f8e-b5b9-8a5cdcdc53d9" containerName="revoke-keys"
Jul 17 20:32:30.842959 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:30.842941    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"
Jul 17 20:32:30.851874 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:30.851843    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"]
Jul 17 20:32:30.955038 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:30.954996    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcb06a3-c0c9-416f-ae88-65e0ccfd4520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-b8dbn\" (UID: \"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"
Jul 17 20:32:30.980461 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:30.980433    2572 scope.go:117] "RemoveContainer" containerID="32cc5439b49a156379a91089d15d8e0092864d6d3614c7cfb05fe36f60d704ac"
Jul 17 20:32:30.980803 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:30.980632    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-grwx2_odh-ai-gateway-infra(5772bde1-4c04-4a8c-85b6-d883e1b54231)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231"
Jul 17 20:32:31.055525 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.055492    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcb06a3-c0c9-416f-ae88-65e0ccfd4520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-b8dbn\" (UID: \"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"
Jul 17 20:32:31.056175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.056152    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcb06a3-c0c9-416f-ae88-65e0ccfd4520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-b8dbn\" (UID: \"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"
Jul 17 20:32:31.153877 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.153775    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"
Jul 17 20:32:31.280964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.280941    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"]
Jul 17 20:32:31.282946 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:32:31.282915    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcb06a3_c0c9_416f_ae88_65e0ccfd4520.slice/crio-4383bddb295ba1b58269bde125368e9d1f35313676210379cc852d17753214e6 WatchSource:0}: Error finding container 4383bddb295ba1b58269bde125368e9d1f35313676210379cc852d17753214e6: Status 404 returned error can't find the container with id 4383bddb295ba1b58269bde125368e9d1f35313676210379cc852d17753214e6
Jul 17 20:32:31.986173 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.986142    2572 generic.go:356] "Generic (PLEG): container finished" podID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerID="ae69a6259b61db0391e22ca117a4c908bc12477d95ff24eca2fa22a3d2d0f40c" exitCode=6
Jul 17 20:32:31.986571 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.986206    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" event={"ID":"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520","Type":"ContainerDied","Data":"ae69a6259b61db0391e22ca117a4c908bc12477d95ff24eca2fa22a3d2d0f40c"}
Jul 17 20:32:31.986571 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.986240    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" event={"ID":"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520","Type":"ContainerStarted","Data":"4383bddb295ba1b58269bde125368e9d1f35313676210379cc852d17753214e6"}
Jul 17 20:32:31.986571 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:31.986382    2572 scope.go:117] "RemoveContainer" containerID="ae69a6259b61db0391e22ca117a4c908bc12477d95ff24eca2fa22a3d2d0f40c"
Jul 17 20:32:32.992040 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:32.992003    2572 generic.go:356] "Generic (PLEG): container finished" podID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerID="12368c806291e44a792eb7b28ba9a1a14bbc973202c8be6ec5924dbf63ea5443" exitCode=6
Jul 17 20:32:32.992474 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:32.992090    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" event={"ID":"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520","Type":"ContainerDied","Data":"12368c806291e44a792eb7b28ba9a1a14bbc973202c8be6ec5924dbf63ea5443"}
Jul 17 20:32:32.992474 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:32.992147    2572 scope.go:117] "RemoveContainer" containerID="ae69a6259b61db0391e22ca117a4c908bc12477d95ff24eca2fa22a3d2d0f40c"
Jul 17 20:32:32.992474 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:32.992301    2572 scope.go:117] "RemoveContainer" containerID="12368c806291e44a792eb7b28ba9a1a14bbc973202c8be6ec5924dbf63ea5443"
Jul 17 20:32:32.992594 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:32.992517    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-b8dbn_odh-ai-gateway-infra(7bcb06a3-c0c9-416f-ae88-65e0ccfd4520)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520"
Jul 17 20:32:33.997805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:33.997778    2572 scope.go:117] "RemoveContainer" containerID="12368c806291e44a792eb7b28ba9a1a14bbc973202c8be6ec5924dbf63ea5443"
Jul 17 20:32:33.998209 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:33.997947    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-b8dbn_odh-ai-gateway-infra(7bcb06a3-c0c9-416f-ae88-65e0ccfd4520)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520"
Jul 17 20:32:38.899490 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:38.899453    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"]
Jul 17 20:32:38.903974 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:38.903958    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"
Jul 17 20:32:38.909809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:38.909781    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"]
Jul 17 20:32:39.022543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:39.022494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5147a4-8899-47df-ba43-1ffd89104312-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc\" (UID: \"fb5147a4-8899-47df-ba43-1ffd89104312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"
Jul 17 20:32:39.123631 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:39.123596    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5147a4-8899-47df-ba43-1ffd89104312-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc\" (UID: \"fb5147a4-8899-47df-ba43-1ffd89104312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"
Jul 17 20:32:39.124219 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:39.124202    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5147a4-8899-47df-ba43-1ffd89104312-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc\" (UID: \"fb5147a4-8899-47df-ba43-1ffd89104312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"
Jul 17 20:32:39.215547 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:39.215522    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"
Jul 17 20:32:39.345996 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:39.345948    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"]
Jul 17 20:32:39.348525 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:32:39.348495    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5147a4_8899_47df_ba43_1ffd89104312.slice/crio-5c83efaab890b965302130fc0d8e1c589f32e563c4730721c162fc84fceba5f6 WatchSource:0}: Error finding container 5c83efaab890b965302130fc0d8e1c589f32e563c4730721c162fc84fceba5f6: Status 404 returned error can't find the container with id 5c83efaab890b965302130fc0d8e1c589f32e563c4730721c162fc84fceba5f6
Jul 17 20:32:40.023736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:40.023706    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5147a4-8899-47df-ba43-1ffd89104312" containerID="1818ef3c78f5c55be0c6ec06554769723ed5189824103b4aa766b4efc3b0efb8" exitCode=6
Jul 17 20:32:40.024128 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:40.023763    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" event={"ID":"fb5147a4-8899-47df-ba43-1ffd89104312","Type":"ContainerDied","Data":"1818ef3c78f5c55be0c6ec06554769723ed5189824103b4aa766b4efc3b0efb8"}
Jul 17 20:32:40.024128 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:40.023799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" event={"ID":"fb5147a4-8899-47df-ba43-1ffd89104312","Type":"ContainerStarted","Data":"5c83efaab890b965302130fc0d8e1c589f32e563c4730721c162fc84fceba5f6"}
Jul 17 20:32:40.024128 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:40.023978    2572 scope.go:117] "RemoveContainer" containerID="1818ef3c78f5c55be0c6ec06554769723ed5189824103b4aa766b4efc3b0efb8"
Jul 17 20:32:41.030370 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:41.030338    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5147a4-8899-47df-ba43-1ffd89104312" containerID="26918b674c66a9c09719fec12763ef14e69a95237fc172f38962bdbb5205a14d" exitCode=6
Jul 17 20:32:41.030868 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:41.030426    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" event={"ID":"fb5147a4-8899-47df-ba43-1ffd89104312","Type":"ContainerDied","Data":"26918b674c66a9c09719fec12763ef14e69a95237fc172f38962bdbb5205a14d"}
Jul 17 20:32:41.030868 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:41.030498    2572 scope.go:117] "RemoveContainer" containerID="1818ef3c78f5c55be0c6ec06554769723ed5189824103b4aa766b4efc3b0efb8"
Jul 17 20:32:41.030868 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:41.030741    2572 scope.go:117] "RemoveContainer" containerID="26918b674c66a9c09719fec12763ef14e69a95237fc172f38962bdbb5205a14d"
Jul 17 20:32:41.030998 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:41.030934    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc_odh-ai-gateway-infra(fb5147a4-8899-47df-ba43-1ffd89104312)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" podUID="fb5147a4-8899-47df-ba43-1ffd89104312"
Jul 17 20:32:42.036615 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:42.036585    2572 scope.go:117] "RemoveContainer" containerID="26918b674c66a9c09719fec12763ef14e69a95237fc172f38962bdbb5205a14d"
Jul 17 20:32:42.036997 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:42.036790    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc_odh-ai-gateway-infra(fb5147a4-8899-47df-ba43-1ffd89104312)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" podUID="fb5147a4-8899-47df-ba43-1ffd89104312"
Jul 17 20:32:43.848805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:43.848773    2572 scope.go:117] "RemoveContainer" containerID="32cc5439b49a156379a91089d15d8e0092864d6d3614c7cfb05fe36f60d704ac"
Jul 17 20:32:45.058014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:45.057982    2572 generic.go:356] "Generic (PLEG): container finished" podID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerID="f17b0e3d3d447a3237b865826cda0604b3d76503b2e83d3fdde609afecd2767b" exitCode=7
Jul 17 20:32:45.058384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:45.058066    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" event={"ID":"5772bde1-4c04-4a8c-85b6-d883e1b54231","Type":"ContainerDied","Data":"f17b0e3d3d447a3237b865826cda0604b3d76503b2e83d3fdde609afecd2767b"}
Jul 17 20:32:45.058384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:45.058145    2572 scope.go:117] "RemoveContainer" containerID="32cc5439b49a156379a91089d15d8e0092864d6d3614c7cfb05fe36f60d704ac"
Jul 17 20:32:45.058495 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:45.058395    2572 scope.go:117] "RemoveContainer" containerID="f17b0e3d3d447a3237b865826cda0604b3d76503b2e83d3fdde609afecd2767b"
Jul 17 20:32:45.058641 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:45.058619    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-grwx2_odh-ai-gateway-infra(5772bde1-4c04-4a8c-85b6-d883e1b54231)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231"
Jul 17 20:32:45.850019 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:45.849993    2572 scope.go:117] "RemoveContainer" containerID="12368c806291e44a792eb7b28ba9a1a14bbc973202c8be6ec5924dbf63ea5443"
Jul 17 20:32:46.082744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:46.082718    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"]
Jul 17 20:32:46.214390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:46.214366    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"
Jul 17 20:32:46.278747 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:46.278718    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5772bde1-4c04-4a8c-85b6-d883e1b54231-maas-api-service-ca\") pod \"5772bde1-4c04-4a8c-85b6-d883e1b54231\" (UID: \"5772bde1-4c04-4a8c-85b6-d883e1b54231\") "
Jul 17 20:32:46.279088 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:46.279043    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772bde1-4c04-4a8c-85b6-d883e1b54231-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5772bde1-4c04-4a8c-85b6-d883e1b54231" (UID: "5772bde1-4c04-4a8c-85b6-d883e1b54231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:32:46.379186 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:46.379151    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5772bde1-4c04-4a8c-85b6-d883e1b54231-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:32:47.071931 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.071897    2572 generic.go:356] "Generic (PLEG): container finished" podID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerID="31138135c969c0c7301a31fa50248d7c56d56f55f0ceb1c2b243598a9114ce63" exitCode=6
Jul 17 20:32:47.072139 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.071968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" event={"ID":"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520","Type":"ContainerDied","Data":"31138135c969c0c7301a31fa50248d7c56d56f55f0ceb1c2b243598a9114ce63"}
Jul 17 20:32:47.072139 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.072028    2572 scope.go:117] "RemoveContainer" containerID="12368c806291e44a792eb7b28ba9a1a14bbc973202c8be6ec5924dbf63ea5443"
Jul 17 20:32:47.072350 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.072331    2572 scope.go:117] "RemoveContainer" containerID="31138135c969c0c7301a31fa50248d7c56d56f55f0ceb1c2b243598a9114ce63"
Jul 17 20:32:47.072583 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:47.072555    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-b8dbn_odh-ai-gateway-infra(7bcb06a3-c0c9-416f-ae88-65e0ccfd4520)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520"
Jul 17 20:32:47.073213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.073190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2" event={"ID":"5772bde1-4c04-4a8c-85b6-d883e1b54231","Type":"ContainerDied","Data":"e328a243118b67f9c91adadda3780b601c7f2235e0763d60d4e518c416369c16"}
Jul 17 20:32:47.073292 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.073222    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"
Jul 17 20:32:47.082707 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.082689    2572 scope.go:117] "RemoveContainer" containerID="f17b0e3d3d447a3237b865826cda0604b3d76503b2e83d3fdde609afecd2767b"
Jul 17 20:32:47.107782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.107758    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"]
Jul 17 20:32:47.113957 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.113930    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-grwx2"]
Jul 17 20:32:47.853575 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:47.853544    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" path="/var/lib/kubelet/pods/5772bde1-4c04-4a8c-85b6-d883e1b54231/volumes"
Jul 17 20:32:48.102776 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:48.102740    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"]
Jul 17 20:32:48.231983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:48.231963    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"
Jul 17 20:32:48.295948 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:48.295918    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcb06a3-c0c9-416f-ae88-65e0ccfd4520-maas-api-service-ca\") pod \"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520\" (UID: \"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520\") "
Jul 17 20:32:48.296325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:48.296300    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcb06a3-c0c9-416f-ae88-65e0ccfd4520-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" (UID: "7bcb06a3-c0c9-416f-ae88-65e0ccfd4520"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:32:48.396734 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:48.396699    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcb06a3-c0c9-416f-ae88-65e0ccfd4520-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:32:49.085014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:49.084981    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"
Jul 17 20:32:49.085203 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:49.085021    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn" event={"ID":"7bcb06a3-c0c9-416f-ae88-65e0ccfd4520","Type":"ContainerDied","Data":"4383bddb295ba1b58269bde125368e9d1f35313676210379cc852d17753214e6"}
Jul 17 20:32:49.085203 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:49.085097    2572 scope.go:117] "RemoveContainer" containerID="31138135c969c0c7301a31fa50248d7c56d56f55f0ceb1c2b243598a9114ce63"
Jul 17 20:32:49.106218 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:49.106198    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"]
Jul 17 20:32:49.108522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:49.108499    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-b8dbn"]
Jul 17 20:32:49.852961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:49.852921    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" path="/var/lib/kubelet/pods/7bcb06a3-c0c9-416f-ae88-65e0ccfd4520/volumes"
Jul 17 20:32:53.848429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:53.848399    2572 scope.go:117] "RemoveContainer" containerID="26918b674c66a9c09719fec12763ef14e69a95237fc172f38962bdbb5205a14d"
Jul 17 20:32:54.109518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:54.109435    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" event={"ID":"fb5147a4-8899-47df-ba43-1ffd89104312","Type":"ContainerStarted","Data":"0c333c7bc91edcb447c1ce29b28398b339084271e4227ed9396041ae8a0b52db"}
Jul 17 20:32:54.125587 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:54.125537    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" podStartSLOduration=16.125521534 podStartE2EDuration="16.125521534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:32:54.124253727 +0000 UTC m=+2208.818229875" watchObservedRunningTime="2026-07-17 20:32:54.125521534 +0000 UTC m=+2208.819497727"
Jul 17 20:32:55.115188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:55.115159    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5147a4-8899-47df-ba43-1ffd89104312" containerID="0c333c7bc91edcb447c1ce29b28398b339084271e4227ed9396041ae8a0b52db" exitCode=6
Jul 17 20:32:55.115602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:55.115230    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" event={"ID":"fb5147a4-8899-47df-ba43-1ffd89104312","Type":"ContainerDied","Data":"0c333c7bc91edcb447c1ce29b28398b339084271e4227ed9396041ae8a0b52db"}
Jul 17 20:32:55.115602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:55.115287    2572 scope.go:117] "RemoveContainer" containerID="26918b674c66a9c09719fec12763ef14e69a95237fc172f38962bdbb5205a14d"
Jul 17 20:32:55.115602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:55.115566    2572 scope.go:117] "RemoveContainer" containerID="0c333c7bc91edcb447c1ce29b28398b339084271e4227ed9396041ae8a0b52db"
Jul 17 20:32:55.115805 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:32:55.115787    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc_odh-ai-gateway-infra(fb5147a4-8899-47df-ba43-1ffd89104312)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" podUID="fb5147a4-8899-47df-ba43-1ffd89104312"
Jul 17 20:32:55.133930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:55.133909    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"]
Jul 17 20:32:56.247373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:56.247342    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"
Jul 17 20:32:56.365190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:56.365160    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5147a4-8899-47df-ba43-1ffd89104312-maas-api-service-ca\") pod \"fb5147a4-8899-47df-ba43-1ffd89104312\" (UID: \"fb5147a4-8899-47df-ba43-1ffd89104312\") "
Jul 17 20:32:56.365558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:56.365533    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5147a4-8899-47df-ba43-1ffd89104312-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb5147a4-8899-47df-ba43-1ffd89104312" (UID: "fb5147a4-8899-47df-ba43-1ffd89104312"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:32:56.466563 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:56.466533    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5147a4-8899-47df-ba43-1ffd89104312-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:32:57.127065 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:57.127026    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"
Jul 17 20:32:57.127256 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:57.127022    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc" event={"ID":"fb5147a4-8899-47df-ba43-1ffd89104312","Type":"ContainerDied","Data":"5c83efaab890b965302130fc0d8e1c589f32e563c4730721c162fc84fceba5f6"}
Jul 17 20:32:57.127256 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:57.127171    2572 scope.go:117] "RemoveContainer" containerID="0c333c7bc91edcb447c1ce29b28398b339084271e4227ed9396041ae8a0b52db"
Jul 17 20:32:57.152275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:57.152246    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"]
Jul 17 20:32:57.154873 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:57.154851    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-8dkbc"]
Jul 17 20:32:57.853116 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:32:57.853085    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" path="/var/lib/kubelet/pods/fb5147a4-8899-47df-ba43-1ffd89104312/volumes"
Jul 17 20:33:20.886664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.886624    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"]
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.886949    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.886960    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.886973    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.886979    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.886987    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.886992    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887001    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887007    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887012    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887017    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887027    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887033    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887041    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887060    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887107    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887114    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887121    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887127    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887132    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887137    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887141    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.887165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.887146    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772bde1-4c04-4a8c-85b6-d883e1b54231" containerName="revoke-keys"
Jul 17 20:33:20.889990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.889974    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"
Jul 17 20:33:20.892389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.892362    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:33:20.892510 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.892410    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:33:20.897482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.897007    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"]
Jul 17 20:33:20.952089 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:20.952027    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fjtwz\" (UID: \"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"
Jul 17 20:33:21.053460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:21.053422    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fjtwz\" (UID: \"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"
Jul 17 20:33:21.054102 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:21.054083    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fjtwz\" (UID: \"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"
Jul 17 20:33:21.200879 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:21.200798    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"
Jul 17 20:33:21.330270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:21.330247    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"]
Jul 17 20:33:21.332582 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:33:21.332555    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1c6845_0c7d_44f2_9a4c_c29c7b0f4d02.slice/crio-0b9fb7efdb65834ce029e80d20285d1fff87b91a3c1a2b904db24efa4af4c938 WatchSource:0}: Error finding container 0b9fb7efdb65834ce029e80d20285d1fff87b91a3c1a2b904db24efa4af4c938: Status 404 returned error can't find the container with id 0b9fb7efdb65834ce029e80d20285d1fff87b91a3c1a2b904db24efa4af4c938
Jul 17 20:33:22.232889 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:22.232860    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerID="08f378e63c80da3d1c395b20067828035d17d3cfc241fe1ec21342eca491477f" exitCode=6
Jul 17 20:33:22.233276 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:22.232935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" event={"ID":"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02","Type":"ContainerDied","Data":"08f378e63c80da3d1c395b20067828035d17d3cfc241fe1ec21342eca491477f"}
Jul 17 20:33:22.233276 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:22.232993    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" event={"ID":"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02","Type":"ContainerStarted","Data":"0b9fb7efdb65834ce029e80d20285d1fff87b91a3c1a2b904db24efa4af4c938"}
Jul 17 20:33:22.233276 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:22.233096    2572 scope.go:117] "RemoveContainer" containerID="08f378e63c80da3d1c395b20067828035d17d3cfc241fe1ec21342eca491477f"
Jul 17 20:33:23.239843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:23.239800    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerID="3229477f607dff887d29f262a7760ec9a367f0db0381bc4a6dad2285848464d9" exitCode=6
Jul 17 20:33:23.240321 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:23.239875    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" event={"ID":"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02","Type":"ContainerDied","Data":"3229477f607dff887d29f262a7760ec9a367f0db0381bc4a6dad2285848464d9"}
Jul 17 20:33:23.240321 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:23.239972    2572 scope.go:117] "RemoveContainer" containerID="08f378e63c80da3d1c395b20067828035d17d3cfc241fe1ec21342eca491477f"
Jul 17 20:33:23.240321 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:23.240232    2572 scope.go:117] "RemoveContainer" containerID="3229477f607dff887d29f262a7760ec9a367f0db0381bc4a6dad2285848464d9"
Jul 17 20:33:23.240469 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:23.240448    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fjtwz_odh-ai-gateway-infra(3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02"
Jul 17 20:33:24.245709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:24.245682    2572 scope.go:117] "RemoveContainer" containerID="3229477f607dff887d29f262a7760ec9a367f0db0381bc4a6dad2285848464d9"
Jul 17 20:33:24.246100 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:24.245846    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fjtwz_odh-ai-gateway-infra(3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02"
Jul 17 20:33:27.098489 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.098446    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"]
Jul 17 20:33:27.098944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.098862    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:27.098944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.098875    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcb06a3-c0c9-416f-ae88-65e0ccfd4520" containerName="revoke-keys"
Jul 17 20:33:27.098944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.098896    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:27.098944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.098901    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:27.099097 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.098960    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5147a4-8899-47df-ba43-1ffd89104312" containerName="revoke-keys"
Jul 17 20:33:27.103032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.103015    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"
Jul 17 20:33:27.110279 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.110257    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"]
Jul 17 20:33:27.199247 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.199213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-pgq64\" (UID: \"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"
Jul 17 20:33:27.300255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.300219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-pgq64\" (UID: \"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"
Jul 17 20:33:27.300879 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.300857    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-pgq64\" (UID: \"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"
Jul 17 20:33:27.414983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.414874    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"
Jul 17 20:33:27.542883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:27.542860    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"]
Jul 17 20:33:27.544973 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:33:27.544949    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fa0d14_e358_4c3d_abcf_ed6f5c2dbdf0.slice/crio-2a69ff2b2314dc9e3e0c77a8ba20e7dfd60a0c293e168597e0f8175ec44cd567 WatchSource:0}: Error finding container 2a69ff2b2314dc9e3e0c77a8ba20e7dfd60a0c293e168597e0f8175ec44cd567: Status 404 returned error can't find the container with id 2a69ff2b2314dc9e3e0c77a8ba20e7dfd60a0c293e168597e0f8175ec44cd567
Jul 17 20:33:28.263228 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:28.263195    2572 generic.go:356] "Generic (PLEG): container finished" podID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerID="8f0c71241793763dc7c0d9d835212ea36a1e1622aca6f85426f0d253b85634f0" exitCode=7
Jul 17 20:33:28.263628 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:28.263277    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" event={"ID":"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0","Type":"ContainerDied","Data":"8f0c71241793763dc7c0d9d835212ea36a1e1622aca6f85426f0d253b85634f0"}
Jul 17 20:33:28.263628 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:28.263340    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" event={"ID":"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0","Type":"ContainerStarted","Data":"2a69ff2b2314dc9e3e0c77a8ba20e7dfd60a0c293e168597e0f8175ec44cd567"}
Jul 17 20:33:28.263628 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:28.263446    2572 scope.go:117] "RemoveContainer" containerID="8f0c71241793763dc7c0d9d835212ea36a1e1622aca6f85426f0d253b85634f0"
Jul 17 20:33:28.947426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:28.947393    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"]
Jul 17 20:33:28.950543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:28.950527    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"
Jul 17 20:33:28.958866 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:28.958841    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"]
Jul 17 20:33:29.014745 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.014711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ee2d60-e53d-4705-a0b0-6a3d86850536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg\" (UID: \"08ee2d60-e53d-4705-a0b0-6a3d86850536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"
Jul 17 20:33:29.116191 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.116097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ee2d60-e53d-4705-a0b0-6a3d86850536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg\" (UID: \"08ee2d60-e53d-4705-a0b0-6a3d86850536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"
Jul 17 20:33:29.116731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.116710    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ee2d60-e53d-4705-a0b0-6a3d86850536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg\" (UID: \"08ee2d60-e53d-4705-a0b0-6a3d86850536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"
Jul 17 20:33:29.262167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.262134    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"
Jul 17 20:33:29.279236 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.279206    2572 generic.go:356] "Generic (PLEG): container finished" podID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerID="55ae146e1b2a0e83c1d3536a95e8c8dd26be10d408d94b425553574732a37ab0" exitCode=7
Jul 17 20:33:29.279701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.279263    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" event={"ID":"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0","Type":"ContainerDied","Data":"55ae146e1b2a0e83c1d3536a95e8c8dd26be10d408d94b425553574732a37ab0"}
Jul 17 20:33:29.279701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.279366    2572 scope.go:117] "RemoveContainer" containerID="8f0c71241793763dc7c0d9d835212ea36a1e1622aca6f85426f0d253b85634f0"
Jul 17 20:33:29.279701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.279655    2572 scope.go:117] "RemoveContainer" containerID="55ae146e1b2a0e83c1d3536a95e8c8dd26be10d408d94b425553574732a37ab0"
Jul 17 20:33:29.279933 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:29.279903    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-pgq64_odh-ai-gateway-infra(e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0"
Jul 17 20:33:29.393553 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:29.393498    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"]
Jul 17 20:33:29.395775 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:33:29.395744    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ee2d60_e53d_4705_a0b0_6a3d86850536.slice/crio-6e066ec4bdbac0fb1fd7a04c20864afb42344e2eb8d57b91620a77626ad4ef1f WatchSource:0}: Error finding container 6e066ec4bdbac0fb1fd7a04c20864afb42344e2eb8d57b91620a77626ad4ef1f: Status 404 returned error can't find the container with id 6e066ec4bdbac0fb1fd7a04c20864afb42344e2eb8d57b91620a77626ad4ef1f
Jul 17 20:33:30.285866 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:30.285778    2572 scope.go:117] "RemoveContainer" containerID="55ae146e1b2a0e83c1d3536a95e8c8dd26be10d408d94b425553574732a37ab0"
Jul 17 20:33:30.286305 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:30.286015    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-pgq64_odh-ai-gateway-infra(e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0"
Jul 17 20:33:30.286991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:30.286967    2572 generic.go:356] "Generic (PLEG): container finished" podID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerID="4484403e5890524a4996ee284d155efbce16d3920e7f26daebbf85c54c2a99e5" exitCode=6
Jul 17 20:33:30.287085 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:30.287063    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" event={"ID":"08ee2d60-e53d-4705-a0b0-6a3d86850536","Type":"ContainerDied","Data":"4484403e5890524a4996ee284d155efbce16d3920e7f26daebbf85c54c2a99e5"}
Jul 17 20:33:30.287136 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:30.287111    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" event={"ID":"08ee2d60-e53d-4705-a0b0-6a3d86850536","Type":"ContainerStarted","Data":"6e066ec4bdbac0fb1fd7a04c20864afb42344e2eb8d57b91620a77626ad4ef1f"}
Jul 17 20:33:30.287204 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:30.287191    2572 scope.go:117] "RemoveContainer" containerID="4484403e5890524a4996ee284d155efbce16d3920e7f26daebbf85c54c2a99e5"
Jul 17 20:33:31.294321 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:31.294293    2572 generic.go:356] "Generic (PLEG): container finished" podID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerID="571e1b91cfc5d3e5020fc9c6f0919a647c568d5db4a07a45e8bf4eff52701acf" exitCode=6
Jul 17 20:33:31.294713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:31.294389    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" event={"ID":"08ee2d60-e53d-4705-a0b0-6a3d86850536","Type":"ContainerDied","Data":"571e1b91cfc5d3e5020fc9c6f0919a647c568d5db4a07a45e8bf4eff52701acf"}
Jul 17 20:33:31.294713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:31.294448    2572 scope.go:117] "RemoveContainer" containerID="4484403e5890524a4996ee284d155efbce16d3920e7f26daebbf85c54c2a99e5"
Jul 17 20:33:31.294713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:31.294564    2572 scope.go:117] "RemoveContainer" containerID="571e1b91cfc5d3e5020fc9c6f0919a647c568d5db4a07a45e8bf4eff52701acf"
Jul 17 20:33:31.294848 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:31.294810    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg_odh-ai-gateway-infra(08ee2d60-e53d-4705-a0b0-6a3d86850536)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536"
Jul 17 20:33:32.302014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:32.301987    2572 scope.go:117] "RemoveContainer" containerID="571e1b91cfc5d3e5020fc9c6f0919a647c568d5db4a07a45e8bf4eff52701acf"
Jul 17 20:33:32.302520 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:32.302303    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg_odh-ai-gateway-infra(08ee2d60-e53d-4705-a0b0-6a3d86850536)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536"
Jul 17 20:33:38.848435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:38.848405    2572 scope.go:117] "RemoveContainer" containerID="3229477f607dff887d29f262a7760ec9a367f0db0381bc4a6dad2285848464d9"
Jul 17 20:33:39.331660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:39.331629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" event={"ID":"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02","Type":"ContainerStarted","Data":"d7f8de218cf5d2876712bd76211e490d1ff202cb7802ad96bde2d8640847a8b3"}
Jul 17 20:33:39.363371 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:39.363319    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" podStartSLOduration=19.363303231 podStartE2EDuration="19.363303231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:33:39.361199806 +0000 UTC m=+2254.055175975" watchObservedRunningTime="2026-07-17 20:33:39.363303231 +0000 UTC m=+2254.057279377"
Jul 17 20:33:40.337764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:40.337734    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerID="d7f8de218cf5d2876712bd76211e490d1ff202cb7802ad96bde2d8640847a8b3" exitCode=6
Jul 17 20:33:40.338268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:40.337802    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" event={"ID":"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02","Type":"ContainerDied","Data":"d7f8de218cf5d2876712bd76211e490d1ff202cb7802ad96bde2d8640847a8b3"}
Jul 17 20:33:40.338268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:40.337858    2572 scope.go:117] "RemoveContainer" containerID="3229477f607dff887d29f262a7760ec9a367f0db0381bc4a6dad2285848464d9"
Jul 17 20:33:40.338268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:40.338120    2572 scope.go:117] "RemoveContainer" containerID="d7f8de218cf5d2876712bd76211e490d1ff202cb7802ad96bde2d8640847a8b3"
Jul 17 20:33:40.338506 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:40.338336    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fjtwz_odh-ai-gateway-infra(3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02"
Jul 17 20:33:40.374621 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:40.374587    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"]
Jul 17 20:33:41.471517 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:41.471492    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"
Jul 17 20:33:41.527914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:41.527885    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02-maas-api-service-ca\") pod \"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02\" (UID: \"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02\") "
Jul 17 20:33:41.528268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:41.528249    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" (UID: "3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:33:41.628653 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:41.628576    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:33:41.848814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:41.848780    2572 scope.go:117] "RemoveContainer" containerID="55ae146e1b2a0e83c1d3536a95e8c8dd26be10d408d94b425553574732a37ab0"
Jul 17 20:33:42.349472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.349445    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"
Jul 17 20:33:42.349642 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.349482    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz" event={"ID":"3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02","Type":"ContainerDied","Data":"0b9fb7efdb65834ce029e80d20285d1fff87b91a3c1a2b904db24efa4af4c938"}
Jul 17 20:33:42.349642 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.349540    2572 scope.go:117] "RemoveContainer" containerID="d7f8de218cf5d2876712bd76211e490d1ff202cb7802ad96bde2d8640847a8b3"
Jul 17 20:33:42.351487 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.351461    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" event={"ID":"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0","Type":"ContainerStarted","Data":"cfff26528bc49dab1962568d187873b0520620b40e0c1711b5b957bfeb972725"}
Jul 17 20:33:42.372028 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.371990    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" podStartSLOduration=15.371976758 podStartE2EDuration="15.371976758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:33:42.371246277 +0000 UTC m=+2257.065222426" watchObservedRunningTime="2026-07-17 20:33:42.371976758 +0000 UTC m=+2257.065952903"
Jul 17 20:33:42.384287 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.384264    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"]
Jul 17 20:33:42.389747 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.389690    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fjtwz"]
Jul 17 20:33:42.848575 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:42.848546    2572 scope.go:117] "RemoveContainer" containerID="571e1b91cfc5d3e5020fc9c6f0919a647c568d5db4a07a45e8bf4eff52701acf"
Jul 17 20:33:43.357944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.357910    2572 generic.go:356] "Generic (PLEG): container finished" podID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerID="cfff26528bc49dab1962568d187873b0520620b40e0c1711b5b957bfeb972725" exitCode=7
Jul 17 20:33:43.358129 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.357979    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" event={"ID":"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0","Type":"ContainerDied","Data":"cfff26528bc49dab1962568d187873b0520620b40e0c1711b5b957bfeb972725"}
Jul 17 20:33:43.358129 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.358035    2572 scope.go:117] "RemoveContainer" containerID="55ae146e1b2a0e83c1d3536a95e8c8dd26be10d408d94b425553574732a37ab0"
Jul 17 20:33:43.358341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.358324    2572 scope.go:117] "RemoveContainer" containerID="cfff26528bc49dab1962568d187873b0520620b40e0c1711b5b957bfeb972725"
Jul 17 20:33:43.358555 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:43.358536    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-pgq64_odh-ai-gateway-infra(e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0"
Jul 17 20:33:43.359696 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.359673    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" event={"ID":"08ee2d60-e53d-4705-a0b0-6a3d86850536","Type":"ContainerStarted","Data":"6dbf5f8c008db7bb87c32f71dd4e7a250fece3fd8198f1d933a6d5d7d8a1b6be"}
Jul 17 20:33:43.380692 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.380641    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"]
Jul 17 20:33:43.401226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.401180    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" podStartSLOduration=15.401167194 podStartE2EDuration="15.401167194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:33:43.394009284 +0000 UTC m=+2258.087985464" watchObservedRunningTime="2026-07-17 20:33:43.401167194 +0000 UTC m=+2258.095143340"
Jul 17 20:33:43.853277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:43.853244    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" path="/var/lib/kubelet/pods/3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02/volumes"
Jul 17 20:33:44.371885 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.371848    2572 generic.go:356] "Generic (PLEG): container finished" podID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerID="6dbf5f8c008db7bb87c32f71dd4e7a250fece3fd8198f1d933a6d5d7d8a1b6be" exitCode=6
Jul 17 20:33:44.372114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.371921    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" event={"ID":"08ee2d60-e53d-4705-a0b0-6a3d86850536","Type":"ContainerDied","Data":"6dbf5f8c008db7bb87c32f71dd4e7a250fece3fd8198f1d933a6d5d7d8a1b6be"}
Jul 17 20:33:44.372114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.372002    2572 scope.go:117] "RemoveContainer" containerID="571e1b91cfc5d3e5020fc9c6f0919a647c568d5db4a07a45e8bf4eff52701acf"
Jul 17 20:33:44.372327 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.372303    2572 scope.go:117] "RemoveContainer" containerID="6dbf5f8c008db7bb87c32f71dd4e7a250fece3fd8198f1d933a6d5d7d8a1b6be"
Jul 17 20:33:44.372537 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:33:44.372515    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg_odh-ai-gateway-infra(08ee2d60-e53d-4705-a0b0-6a3d86850536)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536"
Jul 17 20:33:44.406530 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.406499    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"]
Jul 17 20:33:44.506158 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.506137    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"
Jul 17 20:33:44.655659 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.655575    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0-maas-api-service-ca\") pod \"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0\" (UID: \"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0\") "
Jul 17 20:33:44.655950 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.655929    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" (UID: "e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:33:44.756667 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:44.756630    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:33:45.380970 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.380942    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64" event={"ID":"e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0","Type":"ContainerDied","Data":"2a69ff2b2314dc9e3e0c77a8ba20e7dfd60a0c293e168597e0f8175ec44cd567"}
Jul 17 20:33:45.381375 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.380973    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"
Jul 17 20:33:45.381375 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.380998    2572 scope.go:117] "RemoveContainer" containerID="cfff26528bc49dab1962568d187873b0520620b40e0c1711b5b957bfeb972725"
Jul 17 20:33:45.414661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.414633    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"]
Jul 17 20:33:45.418704 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.418676    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pgq64"]
Jul 17 20:33:45.510090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.510067    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"
Jul 17 20:33:45.562730 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.562696    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ee2d60-e53d-4705-a0b0-6a3d86850536-maas-api-service-ca\") pod \"08ee2d60-e53d-4705-a0b0-6a3d86850536\" (UID: \"08ee2d60-e53d-4705-a0b0-6a3d86850536\") "
Jul 17 20:33:45.563123 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.563094    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ee2d60-e53d-4705-a0b0-6a3d86850536-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08ee2d60-e53d-4705-a0b0-6a3d86850536" (UID: "08ee2d60-e53d-4705-a0b0-6a3d86850536"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:33:45.663312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.663228    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ee2d60-e53d-4705-a0b0-6a3d86850536-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:33:45.854132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:45.854100    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" path="/var/lib/kubelet/pods/e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0/volumes"
Jul 17 20:33:46.386112 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:46.386087    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"
Jul 17 20:33:46.386587 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:46.386112    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg" event={"ID":"08ee2d60-e53d-4705-a0b0-6a3d86850536","Type":"ContainerDied","Data":"6e066ec4bdbac0fb1fd7a04c20864afb42344e2eb8d57b91620a77626ad4ef1f"}
Jul 17 20:33:46.386587 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:46.386167    2572 scope.go:117] "RemoveContainer" containerID="6dbf5f8c008db7bb87c32f71dd4e7a250fece3fd8198f1d933a6d5d7d8a1b6be"
Jul 17 20:33:46.402686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:46.402660    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"]
Jul 17 20:33:46.407104 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:46.407083    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4d4xg"]
Jul 17 20:33:47.853008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:47.852976    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" path="/var/lib/kubelet/pods/08ee2d60-e53d-4705-a0b0-6a3d86850536/volumes"
Jul 17 20:33:49.990850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:33:49.990808    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:34:04.933097 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933041    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"]
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933524    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933543    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933556    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933565    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933580    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933589    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933599    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933607    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933617    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:04.933623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933625    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933646    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933654    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933722    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933733    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933744    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933753    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933760    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933770    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:04.934162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.933778    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:04.938374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.938354    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"
Jul 17 20:34:04.941174 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.941144    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:34:04.941174 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.941157    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:34:04.945632 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:04.945256    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"]
Jul 17 20:34:05.023226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:05.023185    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e3170e-ba1c-405d-8d64-0ac8a07fe66e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd\" (UID: \"93e3170e-ba1c-405d-8d64-0ac8a07fe66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"
Jul 17 20:34:05.124352 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:05.124318    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e3170e-ba1c-405d-8d64-0ac8a07fe66e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd\" (UID: \"93e3170e-ba1c-405d-8d64-0ac8a07fe66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"
Jul 17 20:34:05.124920 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:05.124902    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e3170e-ba1c-405d-8d64-0ac8a07fe66e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd\" (UID: \"93e3170e-ba1c-405d-8d64-0ac8a07fe66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"
Jul 17 20:34:05.250834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:05.250789    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"
Jul 17 20:34:05.376657 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:05.376455    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"]
Jul 17 20:34:05.378976 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:34:05.378946    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e3170e_ba1c_405d_8d64_0ac8a07fe66e.slice/crio-7d29e88355f9c3c6d3a101263ca40f0dfa962a19ee93135850cc4253d1234659 WatchSource:0}: Error finding container 7d29e88355f9c3c6d3a101263ca40f0dfa962a19ee93135850cc4253d1234659: Status 404 returned error can't find the container with id 7d29e88355f9c3c6d3a101263ca40f0dfa962a19ee93135850cc4253d1234659
Jul 17 20:34:05.471398 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:05.471365    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" event={"ID":"93e3170e-ba1c-405d-8d64-0ac8a07fe66e","Type":"ContainerStarted","Data":"7d29e88355f9c3c6d3a101263ca40f0dfa962a19ee93135850cc4253d1234659"}
Jul 17 20:34:06.477352 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:06.477319    2572 generic.go:356] "Generic (PLEG): container finished" podID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerID="a1f69927da8cbd54d9f2f9ad0bb0e17b38eab0eeb957f90047ff9217223f4f62" exitCode=7
Jul 17 20:34:06.477725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:06.477367    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" event={"ID":"93e3170e-ba1c-405d-8d64-0ac8a07fe66e","Type":"ContainerDied","Data":"a1f69927da8cbd54d9f2f9ad0bb0e17b38eab0eeb957f90047ff9217223f4f62"}
Jul 17 20:34:06.477725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:06.477576    2572 scope.go:117] "RemoveContainer" containerID="a1f69927da8cbd54d9f2f9ad0bb0e17b38eab0eeb957f90047ff9217223f4f62"
Jul 17 20:34:07.483566 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:07.483533    2572 generic.go:356] "Generic (PLEG): container finished" podID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerID="7bb9e419a86eeaaf40c4393ab2f12faa12b2be517108344245b34c0a7110f76e" exitCode=7
Jul 17 20:34:07.484039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:07.483593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" event={"ID":"93e3170e-ba1c-405d-8d64-0ac8a07fe66e","Type":"ContainerDied","Data":"7bb9e419a86eeaaf40c4393ab2f12faa12b2be517108344245b34c0a7110f76e"}
Jul 17 20:34:07.484039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:07.483638    2572 scope.go:117] "RemoveContainer" containerID="a1f69927da8cbd54d9f2f9ad0bb0e17b38eab0eeb957f90047ff9217223f4f62"
Jul 17 20:34:07.484039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:07.483837    2572 scope.go:117] "RemoveContainer" containerID="7bb9e419a86eeaaf40c4393ab2f12faa12b2be517108344245b34c0a7110f76e"
Jul 17 20:34:07.484213 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:07.484092    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd_odh-ai-gateway-infra(93e3170e-ba1c-405d-8d64-0ac8a07fe66e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e"
Jul 17 20:34:08.284578 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:08.284533    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:34:08.489665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:08.489642    2572 scope.go:117] "RemoveContainer" containerID="7bb9e419a86eeaaf40c4393ab2f12faa12b2be517108344245b34c0a7110f76e"
Jul 17 20:34:08.490022 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:08.489815    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd_odh-ai-gateway-infra(93e3170e-ba1c-405d-8d64-0ac8a07fe66e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e"
Jul 17 20:34:17.141013 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.140979    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"]
Jul 17 20:34:17.141393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141332    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:17.141393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141343    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:17.141393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141353    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:17.141393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141359    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:17.141393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141373    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:17.141393 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141379    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ee2d60-e53d-4705-a0b0-6a3d86850536" containerName="revoke-keys"
Jul 17 20:34:17.141580 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141422    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1c6845-0c7d-44f2-9a4c-c29c7b0f4d02" containerName="revoke-keys"
Jul 17 20:34:17.141580 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.141431    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fa0d14-e358-4c3d-abcf-ed6f5c2dbdf0" containerName="revoke-keys"
Jul 17 20:34:17.144140 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.144121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"
Jul 17 20:34:17.152396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.152372    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"]
Jul 17 20:34:17.219221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.219188    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93db1f3f-2494-49e5-ba0f-452462204ccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4\" (UID: \"93db1f3f-2494-49e5-ba0f-452462204ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"
Jul 17 20:34:17.319651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.319614    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93db1f3f-2494-49e5-ba0f-452462204ccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4\" (UID: \"93db1f3f-2494-49e5-ba0f-452462204ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"
Jul 17 20:34:17.320207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.320189    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93db1f3f-2494-49e5-ba0f-452462204ccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4\" (UID: \"93db1f3f-2494-49e5-ba0f-452462204ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"
Jul 17 20:34:17.455749 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.455652    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"
Jul 17 20:34:17.586793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:17.586770    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"]
Jul 17 20:34:17.589344 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:34:17.589318    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93db1f3f_2494_49e5_ba0f_452462204ccb.slice/crio-9a0b40853ca134f35c0190a34ef4e542ccee9709a4f90d34b4b6c1b5903ce8fb WatchSource:0}: Error finding container 9a0b40853ca134f35c0190a34ef4e542ccee9709a4f90d34b4b6c1b5903ce8fb: Status 404 returned error can't find the container with id 9a0b40853ca134f35c0190a34ef4e542ccee9709a4f90d34b4b6c1b5903ce8fb
Jul 17 20:34:18.533228 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:18.533133    2572 generic.go:356] "Generic (PLEG): container finished" podID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerID="368d093813da8b6f9e5227bfc155811cda7b87c6c05924e469d0d5c68a707cc4" exitCode=6
Jul 17 20:34:18.533228 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:18.533200    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" event={"ID":"93db1f3f-2494-49e5-ba0f-452462204ccb","Type":"ContainerDied","Data":"368d093813da8b6f9e5227bfc155811cda7b87c6c05924e469d0d5c68a707cc4"}
Jul 17 20:34:18.533648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:18.533251    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" event={"ID":"93db1f3f-2494-49e5-ba0f-452462204ccb","Type":"ContainerStarted","Data":"9a0b40853ca134f35c0190a34ef4e542ccee9709a4f90d34b4b6c1b5903ce8fb"}
Jul 17 20:34:18.533648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:18.533448    2572 scope.go:117] "RemoveContainer" containerID="368d093813da8b6f9e5227bfc155811cda7b87c6c05924e469d0d5c68a707cc4"
Jul 17 20:34:18.994395 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:18.994357    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"]
Jul 17 20:34:18.997682 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:18.997659    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"
Jul 17 20:34:19.006282 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.006259    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"]
Jul 17 20:34:19.034251 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.034227    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73488eb-b5e0-40b3-afbb-44cc0fa0073c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9\" (UID: \"e73488eb-b5e0-40b3-afbb-44cc0fa0073c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"
Jul 17 20:34:19.135125 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.135095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73488eb-b5e0-40b3-afbb-44cc0fa0073c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9\" (UID: \"e73488eb-b5e0-40b3-afbb-44cc0fa0073c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"
Jul 17 20:34:19.135723 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.135699    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73488eb-b5e0-40b3-afbb-44cc0fa0073c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9\" (UID: \"e73488eb-b5e0-40b3-afbb-44cc0fa0073c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"
Jul 17 20:34:19.309631 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.309596    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"
Jul 17 20:34:19.453992 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.453939    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"]
Jul 17 20:34:19.458368 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:34:19.458327    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73488eb_b5e0_40b3_afbb_44cc0fa0073c.slice/crio-a4c89b757dc4e646ed8024516ded6d05d316954c32415ba78d1761522fc08d86 WatchSource:0}: Error finding container a4c89b757dc4e646ed8024516ded6d05d316954c32415ba78d1761522fc08d86: Status 404 returned error can't find the container with id a4c89b757dc4e646ed8024516ded6d05d316954c32415ba78d1761522fc08d86
Jul 17 20:34:19.538474 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.538439    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" event={"ID":"e73488eb-b5e0-40b3-afbb-44cc0fa0073c","Type":"ContainerStarted","Data":"a4c89b757dc4e646ed8024516ded6d05d316954c32415ba78d1761522fc08d86"}
Jul 17 20:34:19.540237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.540208    2572 generic.go:356] "Generic (PLEG): container finished" podID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerID="cd50dfa17aa04a3a93db7a80230d5a3b83016b750b2abc08dce04f449a2fe220" exitCode=6
Jul 17 20:34:19.540367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.540283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" event={"ID":"93db1f3f-2494-49e5-ba0f-452462204ccb","Type":"ContainerDied","Data":"cd50dfa17aa04a3a93db7a80230d5a3b83016b750b2abc08dce04f449a2fe220"}
Jul 17 20:34:19.540367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.540335    2572 scope.go:117] "RemoveContainer" containerID="368d093813da8b6f9e5227bfc155811cda7b87c6c05924e469d0d5c68a707cc4"
Jul 17 20:34:19.540521 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.540507    2572 scope.go:117] "RemoveContainer" containerID="cd50dfa17aa04a3a93db7a80230d5a3b83016b750b2abc08dce04f449a2fe220"
Jul 17 20:34:19.540742 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:19.540722    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4_odh-ai-gateway-infra(93db1f3f-2494-49e5-ba0f-452462204ccb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb"
Jul 17 20:34:19.848863 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:19.848833    2572 scope.go:117] "RemoveContainer" containerID="7bb9e419a86eeaaf40c4393ab2f12faa12b2be517108344245b34c0a7110f76e"
Jul 17 20:34:20.546344 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.546317    2572 generic.go:356] "Generic (PLEG): container finished" podID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerID="9e067d35444f2d24fbf9b4ac9cdd565650438aaf4cca127f5eed650a7ce32190" exitCode=7
Jul 17 20:34:20.546660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.546385    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" event={"ID":"93e3170e-ba1c-405d-8d64-0ac8a07fe66e","Type":"ContainerDied","Data":"9e067d35444f2d24fbf9b4ac9cdd565650438aaf4cca127f5eed650a7ce32190"}
Jul 17 20:34:20.546660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.546489    2572 scope.go:117] "RemoveContainer" containerID="7bb9e419a86eeaaf40c4393ab2f12faa12b2be517108344245b34c0a7110f76e"
Jul 17 20:34:20.546838 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.546818    2572 scope.go:117] "RemoveContainer" containerID="9e067d35444f2d24fbf9b4ac9cdd565650438aaf4cca127f5eed650a7ce32190"
Jul 17 20:34:20.547067 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:20.547029    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd_odh-ai-gateway-infra(93e3170e-ba1c-405d-8d64-0ac8a07fe66e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e"
Jul 17 20:34:20.548078 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.548037    2572 generic.go:356] "Generic (PLEG): container finished" podID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerID="cea2084117b0a48fe455fab76ce311d5dc5cccc7782ebd2964d0952888cf027e" exitCode=6
Jul 17 20:34:20.548188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.548078    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" event={"ID":"e73488eb-b5e0-40b3-afbb-44cc0fa0073c","Type":"ContainerDied","Data":"cea2084117b0a48fe455fab76ce311d5dc5cccc7782ebd2964d0952888cf027e"}
Jul 17 20:34:20.548310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.548295    2572 scope.go:117] "RemoveContainer" containerID="cea2084117b0a48fe455fab76ce311d5dc5cccc7782ebd2964d0952888cf027e"
Jul 17 20:34:20.549676 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.549661    2572 scope.go:117] "RemoveContainer" containerID="cd50dfa17aa04a3a93db7a80230d5a3b83016b750b2abc08dce04f449a2fe220"
Jul 17 20:34:20.549792 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:20.549777    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4_odh-ai-gateway-infra(93db1f3f-2494-49e5-ba0f-452462204ccb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb"
Jul 17 20:34:20.974335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.974301    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"]
Jul 17 20:34:20.977673 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.977652    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"
Jul 17 20:34:20.993343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:20.993315    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"]
Jul 17 20:34:21.049653 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.049612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae2ec39-975b-4b3f-94d0-28f16de72145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-wbn2c\" (UID: \"dae2ec39-975b-4b3f-94d0-28f16de72145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"
Jul 17 20:34:21.151036 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.150951    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae2ec39-975b-4b3f-94d0-28f16de72145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-wbn2c\" (UID: \"dae2ec39-975b-4b3f-94d0-28f16de72145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"
Jul 17 20:34:21.151605 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.151585    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae2ec39-975b-4b3f-94d0-28f16de72145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-wbn2c\" (UID: \"dae2ec39-975b-4b3f-94d0-28f16de72145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"
Jul 17 20:34:21.287505 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.287471    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"
Jul 17 20:34:21.423328 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.423305    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"]
Jul 17 20:34:21.425117 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:34:21.425089    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae2ec39_975b_4b3f_94d0_28f16de72145.slice/crio-6da5e41a42d94701d950e799e8485683fd4979aa410f3668712aa7eb927d0677 WatchSource:0}: Error finding container 6da5e41a42d94701d950e799e8485683fd4979aa410f3668712aa7eb927d0677: Status 404 returned error can't find the container with id 6da5e41a42d94701d950e799e8485683fd4979aa410f3668712aa7eb927d0677
Jul 17 20:34:21.555460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.555426    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" event={"ID":"dae2ec39-975b-4b3f-94d0-28f16de72145","Type":"ContainerStarted","Data":"6da5e41a42d94701d950e799e8485683fd4979aa410f3668712aa7eb927d0677"}
Jul 17 20:34:21.558564 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.558540    2572 generic.go:356] "Generic (PLEG): container finished" podID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerID="9db9136eb8ebe5bcc19a97e70113c4faf5f7c525cfcd0e73b1b462bfee1755ad" exitCode=6
Jul 17 20:34:21.558675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.558580    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" event={"ID":"e73488eb-b5e0-40b3-afbb-44cc0fa0073c","Type":"ContainerDied","Data":"9db9136eb8ebe5bcc19a97e70113c4faf5f7c525cfcd0e73b1b462bfee1755ad"}
Jul 17 20:34:21.558675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.558621    2572 scope.go:117] "RemoveContainer" containerID="cea2084117b0a48fe455fab76ce311d5dc5cccc7782ebd2964d0952888cf027e"
Jul 17 20:34:21.558880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.558849    2572 scope.go:117] "RemoveContainer" containerID="9db9136eb8ebe5bcc19a97e70113c4faf5f7c525cfcd0e73b1b462bfee1755ad"
Jul 17 20:34:21.559072 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:21.559035    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9_odh-ai-gateway-infra(e73488eb-b5e0-40b3-afbb-44cc0fa0073c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c"
Jul 17 20:34:21.573763 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.573721    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"]
Jul 17 20:34:21.708568 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.708546    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"
Jul 17 20:34:21.754793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.754768    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e3170e-ba1c-405d-8d64-0ac8a07fe66e-maas-api-service-ca\") pod \"93e3170e-ba1c-405d-8d64-0ac8a07fe66e\" (UID: \"93e3170e-ba1c-405d-8d64-0ac8a07fe66e\") "
Jul 17 20:34:21.755115 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.755094    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e3170e-ba1c-405d-8d64-0ac8a07fe66e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93e3170e-ba1c-405d-8d64-0ac8a07fe66e" (UID: "93e3170e-ba1c-405d-8d64-0ac8a07fe66e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:34:21.860827 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:21.855975    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e3170e-ba1c-405d-8d64-0ac8a07fe66e-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:34:22.564334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.564301    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"
Jul 17 20:34:22.564334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.564309    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd" event={"ID":"93e3170e-ba1c-405d-8d64-0ac8a07fe66e","Type":"ContainerDied","Data":"7d29e88355f9c3c6d3a101263ca40f0dfa962a19ee93135850cc4253d1234659"}
Jul 17 20:34:22.564809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.564403    2572 scope.go:117] "RemoveContainer" containerID="9e067d35444f2d24fbf9b4ac9cdd565650438aaf4cca127f5eed650a7ce32190"
Jul 17 20:34:22.566099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.566081    2572 scope.go:117] "RemoveContainer" containerID="9db9136eb8ebe5bcc19a97e70113c4faf5f7c525cfcd0e73b1b462bfee1755ad"
Jul 17 20:34:22.566295 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:22.566270    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9_odh-ai-gateway-infra(e73488eb-b5e0-40b3-afbb-44cc0fa0073c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c"
Jul 17 20:34:22.567641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.567616    2572 generic.go:356] "Generic (PLEG): container finished" podID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerID="04995c70e48f1d11fa002505bd9fc12e963c6f1c8908175d98c70da33d792ff0" exitCode=6
Jul 17 20:34:22.567729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.567661    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" event={"ID":"dae2ec39-975b-4b3f-94d0-28f16de72145","Type":"ContainerDied","Data":"04995c70e48f1d11fa002505bd9fc12e963c6f1c8908175d98c70da33d792ff0"}
Jul 17 20:34:22.567829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.567817    2572 scope.go:117] "RemoveContainer" containerID="04995c70e48f1d11fa002505bd9fc12e963c6f1c8908175d98c70da33d792ff0"
Jul 17 20:34:22.619444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.619419    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"]
Jul 17 20:34:22.623684 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:22.623660    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-n4qzd"]
Jul 17 20:34:23.573848 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:23.573814    2572 generic.go:356] "Generic (PLEG): container finished" podID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerID="ad1c6de55621ca8a358a8b22a01cdb5df1a4c5f15935da38f49e8fe6d892ec0a" exitCode=6
Jul 17 20:34:23.574338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:23.573899    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" event={"ID":"dae2ec39-975b-4b3f-94d0-28f16de72145","Type":"ContainerDied","Data":"ad1c6de55621ca8a358a8b22a01cdb5df1a4c5f15935da38f49e8fe6d892ec0a"}
Jul 17 20:34:23.574338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:23.573960    2572 scope.go:117] "RemoveContainer" containerID="04995c70e48f1d11fa002505bd9fc12e963c6f1c8908175d98c70da33d792ff0"
Jul 17 20:34:23.574338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:23.574144    2572 scope.go:117] "RemoveContainer" containerID="ad1c6de55621ca8a358a8b22a01cdb5df1a4c5f15935da38f49e8fe6d892ec0a"
Jul 17 20:34:23.574497 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:23.574343    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-wbn2c_odh-ai-gateway-infra(dae2ec39-975b-4b3f-94d0-28f16de72145)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145"
Jul 17 20:34:23.854268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:23.854189    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" path="/var/lib/kubelet/pods/93e3170e-ba1c-405d-8d64-0ac8a07fe66e/volumes"
Jul 17 20:34:24.581218 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:24.581193    2572 scope.go:117] "RemoveContainer" containerID="ad1c6de55621ca8a358a8b22a01cdb5df1a4c5f15935da38f49e8fe6d892ec0a"
Jul 17 20:34:24.581592 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:24.581353    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-wbn2c_odh-ai-gateway-infra(dae2ec39-975b-4b3f-94d0-28f16de72145)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145"
Jul 17 20:34:33.847840 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:33.847811    2572 scope.go:117] "RemoveContainer" containerID="9db9136eb8ebe5bcc19a97e70113c4faf5f7c525cfcd0e73b1b462bfee1755ad"
Jul 17 20:34:34.623940 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:34.623909    2572 generic.go:356] "Generic (PLEG): container finished" podID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerID="ed3122b38c0405d6b8080171696869998f8cf112f11e5ea73acc6ef6b6e71a9f" exitCode=6
Jul 17 20:34:34.624130 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:34.623947    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" event={"ID":"e73488eb-b5e0-40b3-afbb-44cc0fa0073c","Type":"ContainerDied","Data":"ed3122b38c0405d6b8080171696869998f8cf112f11e5ea73acc6ef6b6e71a9f"}
Jul 17 20:34:34.624130 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:34.624009    2572 scope.go:117] "RemoveContainer" containerID="9db9136eb8ebe5bcc19a97e70113c4faf5f7c525cfcd0e73b1b462bfee1755ad"
Jul 17 20:34:34.624322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:34.624303    2572 scope.go:117] "RemoveContainer" containerID="ed3122b38c0405d6b8080171696869998f8cf112f11e5ea73acc6ef6b6e71a9f"
Jul 17 20:34:34.624546 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:34.624518    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9_odh-ai-gateway-infra(e73488eb-b5e0-40b3-afbb-44cc0fa0073c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c"
Jul 17 20:34:34.847787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:34.847754    2572 scope.go:117] "RemoveContainer" containerID="cd50dfa17aa04a3a93db7a80230d5a3b83016b750b2abc08dce04f449a2fe220"
Jul 17 20:34:35.630893 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.630860    2572 generic.go:356] "Generic (PLEG): container finished" podID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerID="d090b76ab4bd92a0c1b1f55874481a8d1ca6ebe41fe5375bd9ccd97fbf984049" exitCode=6
Jul 17 20:34:35.631376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.630934    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" event={"ID":"93db1f3f-2494-49e5-ba0f-452462204ccb","Type":"ContainerDied","Data":"d090b76ab4bd92a0c1b1f55874481a8d1ca6ebe41fe5375bd9ccd97fbf984049"}
Jul 17 20:34:35.631376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.631018    2572 scope.go:117] "RemoveContainer" containerID="cd50dfa17aa04a3a93db7a80230d5a3b83016b750b2abc08dce04f449a2fe220"
Jul 17 20:34:35.631376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.631327    2572 scope.go:117] "RemoveContainer" containerID="d090b76ab4bd92a0c1b1f55874481a8d1ca6ebe41fe5375bd9ccd97fbf984049"
Jul 17 20:34:35.631570 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:35.631550    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4_odh-ai-gateway-infra(93db1f3f-2494-49e5-ba0f-452462204ccb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb"
Jul 17 20:34:35.651607 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.651582    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"]
Jul 17 20:34:35.782804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.782774    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"
Jul 17 20:34:35.849855 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.849830    2572 scope.go:117] "RemoveContainer" containerID="ad1c6de55621ca8a358a8b22a01cdb5df1a4c5f15935da38f49e8fe6d892ec0a"
Jul 17 20:34:35.868574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.868554    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73488eb-b5e0-40b3-afbb-44cc0fa0073c-maas-api-service-ca\") pod \"e73488eb-b5e0-40b3-afbb-44cc0fa0073c\" (UID: \"e73488eb-b5e0-40b3-afbb-44cc0fa0073c\") "
Jul 17 20:34:35.868909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.868890    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73488eb-b5e0-40b3-afbb-44cc0fa0073c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e73488eb-b5e0-40b3-afbb-44cc0fa0073c" (UID: "e73488eb-b5e0-40b3-afbb-44cc0fa0073c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:34:35.969769 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:35.969734    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73488eb-b5e0-40b3-afbb-44cc0fa0073c-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:34:36.638831 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.638795    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"
Jul 17 20:34:36.639295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.638795    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9" event={"ID":"e73488eb-b5e0-40b3-afbb-44cc0fa0073c","Type":"ContainerDied","Data":"a4c89b757dc4e646ed8024516ded6d05d316954c32415ba78d1761522fc08d86"}
Jul 17 20:34:36.639295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.638952    2572 scope.go:117] "RemoveContainer" containerID="ed3122b38c0405d6b8080171696869998f8cf112f11e5ea73acc6ef6b6e71a9f"
Jul 17 20:34:36.640684 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.640656    2572 generic.go:356] "Generic (PLEG): container finished" podID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerID="997f7bf83746670f172a9204d72c2c47eb3b6e1ee53e355d79c3e756c6500c2c" exitCode=6
Jul 17 20:34:36.640782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.640694    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" event={"ID":"dae2ec39-975b-4b3f-94d0-28f16de72145","Type":"ContainerDied","Data":"997f7bf83746670f172a9204d72c2c47eb3b6e1ee53e355d79c3e756c6500c2c"}
Jul 17 20:34:36.641021 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.641005    2572 scope.go:117] "RemoveContainer" containerID="997f7bf83746670f172a9204d72c2c47eb3b6e1ee53e355d79c3e756c6500c2c"
Jul 17 20:34:36.641254 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:36.641234    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-wbn2c_odh-ai-gateway-infra(dae2ec39-975b-4b3f-94d0-28f16de72145)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145"
Jul 17 20:34:36.647932 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.647808    2572 scope.go:117] "RemoveContainer" containerID="ad1c6de55621ca8a358a8b22a01cdb5df1a4c5f15935da38f49e8fe6d892ec0a"
Jul 17 20:34:36.657690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.657670    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"]
Jul 17 20:34:36.680106 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.680072    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"]
Jul 17 20:34:36.690842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.690816    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-59vs9"]
Jul 17 20:34:36.787438 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.787380    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"
Jul 17 20:34:36.876842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.876801    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93db1f3f-2494-49e5-ba0f-452462204ccb-maas-api-service-ca\") pod \"93db1f3f-2494-49e5-ba0f-452462204ccb\" (UID: \"93db1f3f-2494-49e5-ba0f-452462204ccb\") "
Jul 17 20:34:36.877245 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.877221    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93db1f3f-2494-49e5-ba0f-452462204ccb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93db1f3f-2494-49e5-ba0f-452462204ccb" (UID: "93db1f3f-2494-49e5-ba0f-452462204ccb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:34:36.978295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:36.978251    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93db1f3f-2494-49e5-ba0f-452462204ccb-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:34:37.648238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.648201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4" event={"ID":"93db1f3f-2494-49e5-ba0f-452462204ccb","Type":"ContainerDied","Data":"9a0b40853ca134f35c0190a34ef4e542ccee9709a4f90d34b4b6c1b5903ce8fb"}
Jul 17 20:34:37.648238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.648238    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"
Jul 17 20:34:37.648701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.648268    2572 scope.go:117] "RemoveContainer" containerID="d090b76ab4bd92a0c1b1f55874481a8d1ca6ebe41fe5375bd9ccd97fbf984049"
Jul 17 20:34:37.676876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.676852    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"]
Jul 17 20:34:37.681254 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.681234    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"]
Jul 17 20:34:37.685991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.685966    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nlfr4"]
Jul 17 20:34:37.807288 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.807262    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"
Jul 17 20:34:37.853613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.853579    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" path="/var/lib/kubelet/pods/93db1f3f-2494-49e5-ba0f-452462204ccb/volumes"
Jul 17 20:34:37.853904 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.853891    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" path="/var/lib/kubelet/pods/e73488eb-b5e0-40b3-afbb-44cc0fa0073c/volumes"
Jul 17 20:34:37.886026 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.885999    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae2ec39-975b-4b3f-94d0-28f16de72145-maas-api-service-ca\") pod \"dae2ec39-975b-4b3f-94d0-28f16de72145\" (UID: \"dae2ec39-975b-4b3f-94d0-28f16de72145\") "
Jul 17 20:34:37.886352 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.886334    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae2ec39-975b-4b3f-94d0-28f16de72145-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dae2ec39-975b-4b3f-94d0-28f16de72145" (UID: "dae2ec39-975b-4b3f-94d0-28f16de72145"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:34:37.987024 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:37.986997    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae2ec39-975b-4b3f-94d0-28f16de72145-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:34:38.654488 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:38.654455    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"
Jul 17 20:34:38.655014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:38.654454    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c" event={"ID":"dae2ec39-975b-4b3f-94d0-28f16de72145","Type":"ContainerDied","Data":"6da5e41a42d94701d950e799e8485683fd4979aa410f3668712aa7eb927d0677"}
Jul 17 20:34:38.655014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:38.654583    2572 scope.go:117] "RemoveContainer" containerID="997f7bf83746670f172a9204d72c2c47eb3b6e1ee53e355d79c3e756c6500c2c"
Jul 17 20:34:38.679518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:38.679492    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"]
Jul 17 20:34:38.684596 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:38.684571    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-wbn2c"]
Jul 17 20:34:39.852652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:39.852614    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" path="/var/lib/kubelet/pods/dae2ec39-975b-4b3f-94d0-28f16de72145/volumes"
Jul 17 20:34:54.979947 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.979912    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"]
Jul 17 20:34:54.980444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980393    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:34:54.980444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980412    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:34:54.980444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980429    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:34:54.980444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980438    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980448    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980457    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980472    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980480    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980495    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980503    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980513    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980522    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980531    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980539    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980550    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980558    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980570    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980579    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980660    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980671    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:34:54.980679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980680    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:34:54.981486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980689    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:34:54.981486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980698    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:34:54.981486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980705    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:34:54.981486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980713    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:34:54.981486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980722    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:34:54.981486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.980730    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:34:54.984084 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.984065    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"
Jul 17 20:34:54.986760 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.986739    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:34:54.986865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.986848    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:34:54.991441 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:54.990975    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"]
Jul 17 20:34:55.131819 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:55.131783    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-h7snt\" (UID: \"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"
Jul 17 20:34:55.232843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:55.232754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-h7snt\" (UID: \"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"
Jul 17 20:34:55.233396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:55.233378    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-h7snt\" (UID: \"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"
Jul 17 20:34:55.294865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:55.294822    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"
Jul 17 20:34:55.426196 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:55.426168    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"]
Jul 17 20:34:55.429258 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:34:55.429236    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a69bf7e_c56e_456b_8bcd_56f7e2d2e93e.slice/crio-d506ba6987b4427d6f303fe07b128d298e698d2a9e2f6176f45f0f5cb37c848b WatchSource:0}: Error finding container d506ba6987b4427d6f303fe07b128d298e698d2a9e2f6176f45f0f5cb37c848b: Status 404 returned error can't find the container with id d506ba6987b4427d6f303fe07b128d298e698d2a9e2f6176f45f0f5cb37c848b
Jul 17 20:34:55.731817 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:55.731790    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" event={"ID":"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e","Type":"ContainerStarted","Data":"d506ba6987b4427d6f303fe07b128d298e698d2a9e2f6176f45f0f5cb37c848b"}
Jul 17 20:34:56.737740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:56.737704    2572 generic.go:356] "Generic (PLEG): container finished" podID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerID="451e69a61c1badf3e467425012b8fc564bdca5f8f022208b6d5a6fb97592962f" exitCode=7
Jul 17 20:34:56.738134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:56.737793    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" event={"ID":"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e","Type":"ContainerDied","Data":"451e69a61c1badf3e467425012b8fc564bdca5f8f022208b6d5a6fb97592962f"}
Jul 17 20:34:56.738134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:56.737942    2572 scope.go:117] "RemoveContainer" containerID="451e69a61c1badf3e467425012b8fc564bdca5f8f022208b6d5a6fb97592962f"
Jul 17 20:34:57.743853 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:57.743818    2572 generic.go:356] "Generic (PLEG): container finished" podID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerID="567a22466791391aae67348a70f1bded6abb6da520ad12ebc033f1a644fc5c82" exitCode=7
Jul 17 20:34:57.744339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:57.743899    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" event={"ID":"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e","Type":"ContainerDied","Data":"567a22466791391aae67348a70f1bded6abb6da520ad12ebc033f1a644fc5c82"}
Jul 17 20:34:57.744339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:57.743967    2572 scope.go:117] "RemoveContainer" containerID="451e69a61c1badf3e467425012b8fc564bdca5f8f022208b6d5a6fb97592962f"
Jul 17 20:34:57.744339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:57.744174    2572 scope.go:117] "RemoveContainer" containerID="567a22466791391aae67348a70f1bded6abb6da520ad12ebc033f1a644fc5c82"
Jul 17 20:34:57.744488 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:57.744344    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-h7snt_odh-ai-gateway-infra(0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e"
Jul 17 20:34:58.750307 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:34:58.750276    2572 scope.go:117] "RemoveContainer" containerID="567a22466791391aae67348a70f1bded6abb6da520ad12ebc033f1a644fc5c82"
Jul 17 20:34:58.750783 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:34:58.750537    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-h7snt_odh-ai-gateway-infra(0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e"
Jul 17 20:35:09.067837 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.067803    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"]
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068117    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068130    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae2ec39-975b-4b3f-94d0-28f16de72145" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068140    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068145    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068159    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068165    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068217    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="93db1f3f-2494-49e5-ba0f-452462204ccb" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068231    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73488eb-b5e0-40b3-afbb-44cc0fa0073c" containerName="revoke-keys"
Jul 17 20:35:09.068295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.068237    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e3170e-ba1c-405d-8d64-0ac8a07fe66e" containerName="revoke-keys"
Jul 17 20:35:09.072579 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.072560    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"
Jul 17 20:35:09.087061 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.086829    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"]
Jul 17 20:35:09.243853 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.243804    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a263319-6bb0-4ad1-a589-36e2e612c8a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh\" (UID: \"0a263319-6bb0-4ad1-a589-36e2e612c8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"
Jul 17 20:35:09.344481 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.344398    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a263319-6bb0-4ad1-a589-36e2e612c8a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh\" (UID: \"0a263319-6bb0-4ad1-a589-36e2e612c8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"
Jul 17 20:35:09.345091 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.345072    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a263319-6bb0-4ad1-a589-36e2e612c8a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh\" (UID: \"0a263319-6bb0-4ad1-a589-36e2e612c8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"
Jul 17 20:35:09.382614 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.382589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"
Jul 17 20:35:09.520310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.520288    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"]
Jul 17 20:35:09.522802 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:35:09.522776    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a263319_6bb0_4ad1_a589_36e2e612c8a7.slice/crio-2289a27f02e37f95dbe5e29b51eb4c7a8e18dfe75ec059479b6e40c45c0606b0 WatchSource:0}: Error finding container 2289a27f02e37f95dbe5e29b51eb4c7a8e18dfe75ec059479b6e40c45c0606b0: Status 404 returned error can't find the container with id 2289a27f02e37f95dbe5e29b51eb4c7a8e18dfe75ec059479b6e40c45c0606b0
Jul 17 20:35:09.798780 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:09.798748    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" event={"ID":"0a263319-6bb0-4ad1-a589-36e2e612c8a7","Type":"ContainerStarted","Data":"2289a27f02e37f95dbe5e29b51eb4c7a8e18dfe75ec059479b6e40c45c0606b0"}
Jul 17 20:35:10.809402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:10.809359    2572 generic.go:356] "Generic (PLEG): container finished" podID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerID="c7bf38e0f0495eb774466683574038531644b854ff72bf0c2f45470a5c6cdf51" exitCode=6
Jul 17 20:35:10.809810 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:10.809443    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" event={"ID":"0a263319-6bb0-4ad1-a589-36e2e612c8a7","Type":"ContainerDied","Data":"c7bf38e0f0495eb774466683574038531644b854ff72bf0c2f45470a5c6cdf51"}
Jul 17 20:35:10.809810 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:10.809584    2572 scope.go:117] "RemoveContainer" containerID="c7bf38e0f0495eb774466683574038531644b854ff72bf0c2f45470a5c6cdf51"
Jul 17 20:35:11.007230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.007195    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"]
Jul 17 20:35:11.011482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.011466    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"
Jul 17 20:35:11.022624 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.022599    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"]
Jul 17 20:35:11.157616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.157511    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac30027b-4ab8-433b-ab43-8e3768cf0baf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-6jcfr\" (UID: \"ac30027b-4ab8-433b-ab43-8e3768cf0baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"
Jul 17 20:35:11.258480 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.258445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac30027b-4ab8-433b-ab43-8e3768cf0baf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-6jcfr\" (UID: \"ac30027b-4ab8-433b-ab43-8e3768cf0baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"
Jul 17 20:35:11.259039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.259021    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac30027b-4ab8-433b-ab43-8e3768cf0baf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-6jcfr\" (UID: \"ac30027b-4ab8-433b-ab43-8e3768cf0baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"
Jul 17 20:35:11.321536 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.321492    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"
Jul 17 20:35:11.466536 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.466508    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"]
Jul 17 20:35:11.505936 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:35:11.505903    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac30027b_4ab8_433b_ab43_8e3768cf0baf.slice/crio-e0fe274ca5a5c4a5a05b6782339eb40e8e5326ec4dfb97c0bd8e39086bcd5c81 WatchSource:0}: Error finding container e0fe274ca5a5c4a5a05b6782339eb40e8e5326ec4dfb97c0bd8e39086bcd5c81: Status 404 returned error can't find the container with id e0fe274ca5a5c4a5a05b6782339eb40e8e5326ec4dfb97c0bd8e39086bcd5c81
Jul 17 20:35:11.815629 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.815600    2572 generic.go:356] "Generic (PLEG): container finished" podID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerID="e65e3a946236feed976372282a030bb8ecd860048525c5962b89446f120c0091" exitCode=6
Jul 17 20:35:11.816107 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.815671    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" event={"ID":"0a263319-6bb0-4ad1-a589-36e2e612c8a7","Type":"ContainerDied","Data":"e65e3a946236feed976372282a030bb8ecd860048525c5962b89446f120c0091"}
Jul 17 20:35:11.816107 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.815740    2572 scope.go:117] "RemoveContainer" containerID="c7bf38e0f0495eb774466683574038531644b854ff72bf0c2f45470a5c6cdf51"
Jul 17 20:35:11.816107 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.815962    2572 scope.go:117] "RemoveContainer" containerID="e65e3a946236feed976372282a030bb8ecd860048525c5962b89446f120c0091"
Jul 17 20:35:11.816290 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:11.816202    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh_odh-ai-gateway-infra(0a263319-6bb0-4ad1-a589-36e2e612c8a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7"
Jul 17 20:35:11.817414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.817394    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" event={"ID":"ac30027b-4ab8-433b-ab43-8e3768cf0baf","Type":"ContainerStarted","Data":"5dbf9ba2c5048cacbef39a5835339eff0afeedb008c53cf0b98c8a9acbdbfa77"}
Jul 17 20:35:11.817493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.817431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" event={"ID":"ac30027b-4ab8-433b-ab43-8e3768cf0baf","Type":"ContainerStarted","Data":"e0fe274ca5a5c4a5a05b6782339eb40e8e5326ec4dfb97c0bd8e39086bcd5c81"}
Jul 17 20:35:11.847739 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.847718    2572 scope.go:117] "RemoveContainer" containerID="567a22466791391aae67348a70f1bded6abb6da520ad12ebc033f1a644fc5c82"
Jul 17 20:35:11.880442 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:11.880394    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" podStartSLOduration=1.880378645 podStartE2EDuration="1.880378645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:35:11.878742135 +0000 UTC m=+2346.572718292" watchObservedRunningTime="2026-07-17 20:35:11.880378645 +0000 UTC m=+2346.574354790"
Jul 17 20:35:12.823249 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.823217    2572 scope.go:117] "RemoveContainer" containerID="e65e3a946236feed976372282a030bb8ecd860048525c5962b89446f120c0091"
Jul 17 20:35:12.823696 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:12.823452    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh_odh-ai-gateway-infra(0a263319-6bb0-4ad1-a589-36e2e612c8a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7"
Jul 17 20:35:12.824763 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.824742    2572 generic.go:356] "Generic (PLEG): container finished" podID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerID="f9959cb2d28ad3f35a24570284922db1722cfdb64106e1f2cae8279435518055" exitCode=7
Jul 17 20:35:12.824828 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.824810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" event={"ID":"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e","Type":"ContainerDied","Data":"f9959cb2d28ad3f35a24570284922db1722cfdb64106e1f2cae8279435518055"}
Jul 17 20:35:12.824877 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.824861    2572 scope.go:117] "RemoveContainer" containerID="567a22466791391aae67348a70f1bded6abb6da520ad12ebc033f1a644fc5c82"
Jul 17 20:35:12.825182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.825169    2572 scope.go:117] "RemoveContainer" containerID="f9959cb2d28ad3f35a24570284922db1722cfdb64106e1f2cae8279435518055"
Jul 17 20:35:12.825370 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:12.825347    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-h7snt_odh-ai-gateway-infra(0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e"
Jul 17 20:35:12.826340 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.826319    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerID="5dbf9ba2c5048cacbef39a5835339eff0afeedb008c53cf0b98c8a9acbdbfa77" exitCode=6
Jul 17 20:35:12.826427 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.826353    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" event={"ID":"ac30027b-4ab8-433b-ab43-8e3768cf0baf","Type":"ContainerDied","Data":"5dbf9ba2c5048cacbef39a5835339eff0afeedb008c53cf0b98c8a9acbdbfa77"}
Jul 17 20:35:12.826522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:12.826509    2572 scope.go:117] "RemoveContainer" containerID="5dbf9ba2c5048cacbef39a5835339eff0afeedb008c53cf0b98c8a9acbdbfa77"
Jul 17 20:35:13.833887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:13.833850    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerID="c69b23c237b5f4bbcec873ad9ee38d21424f7c7d09b8b54d94e96db2e86b1c10" exitCode=6
Jul 17 20:35:13.834329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:13.833935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" event={"ID":"ac30027b-4ab8-433b-ab43-8e3768cf0baf","Type":"ContainerDied","Data":"c69b23c237b5f4bbcec873ad9ee38d21424f7c7d09b8b54d94e96db2e86b1c10"}
Jul 17 20:35:13.834329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:13.833993    2572 scope.go:117] "RemoveContainer" containerID="5dbf9ba2c5048cacbef39a5835339eff0afeedb008c53cf0b98c8a9acbdbfa77"
Jul 17 20:35:13.834329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:13.834217    2572 scope.go:117] "RemoveContainer" containerID="c69b23c237b5f4bbcec873ad9ee38d21424f7c7d09b8b54d94e96db2e86b1c10"
Jul 17 20:35:13.834450 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:13.834413    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-6jcfr_odh-ai-gateway-infra(ac30027b-4ab8-433b-ab43-8e3768cf0baf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf"
Jul 17 20:35:13.879924 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:13.879890    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"]
Jul 17 20:35:14.009406 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.009384    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"
Jul 17 20:35:14.184414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.184385    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e-maas-api-service-ca\") pod \"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e\" (UID: \"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e\") "
Jul 17 20:35:14.184746 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.184721    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" (UID: "0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:35:14.285864 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.285825    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:35:14.839247 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.839217    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"
Jul 17 20:35:14.839674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.839223    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt" event={"ID":"0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e","Type":"ContainerDied","Data":"d506ba6987b4427d6f303fe07b128d298e698d2a9e2f6176f45f0f5cb37c848b"}
Jul 17 20:35:14.839674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.839368    2572 scope.go:117] "RemoveContainer" containerID="f9959cb2d28ad3f35a24570284922db1722cfdb64106e1f2cae8279435518055"
Jul 17 20:35:14.841215 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.841197    2572 scope.go:117] "RemoveContainer" containerID="c69b23c237b5f4bbcec873ad9ee38d21424f7c7d09b8b54d94e96db2e86b1c10"
Jul 17 20:35:14.841429 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:14.841405    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-6jcfr_odh-ai-gateway-infra(ac30027b-4ab8-433b-ab43-8e3768cf0baf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf"
Jul 17 20:35:14.877014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.876983    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"]
Jul 17 20:35:14.883467 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:14.883440    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-h7snt"]
Jul 17 20:35:15.854321 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:15.854286    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" path="/var/lib/kubelet/pods/0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e/volumes"
Jul 17 20:35:17.200610 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.200578    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"]
Jul 17 20:35:17.200976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.200889    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:17.200976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.200900    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:17.200976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.200906    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:17.200976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.200911    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:17.200976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.200974    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:17.201184 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.200986    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:17.205096 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.205075    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"
Jul 17 20:35:17.212259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.212235    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"]
Jul 17 20:35:17.313629 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.313596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rs57h\" (UID: \"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"
Jul 17 20:35:17.414072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.414012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rs57h\" (UID: \"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"
Jul 17 20:35:17.414765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.414746    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rs57h\" (UID: \"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"
Jul 17 20:35:17.515910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.515879    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"
Jul 17 20:35:17.658584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.658560    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"]
Jul 17 20:35:17.661021 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:35:17.660987    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bc33ca_bde0_44bb_9ea1_dc197c5dc3de.slice/crio-4a5100cd23fa77ddb9ff759040650e9bd600f53411e61c316dc413779df262f5 WatchSource:0}: Error finding container 4a5100cd23fa77ddb9ff759040650e9bd600f53411e61c316dc413779df262f5: Status 404 returned error can't find the container with id 4a5100cd23fa77ddb9ff759040650e9bd600f53411e61c316dc413779df262f5
Jul 17 20:35:17.855537 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.855505    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" event={"ID":"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de","Type":"ContainerStarted","Data":"62b378384d1533ecf19cb38c15b02a6710195c55734fe569d0e80598eebcfea6"}
Jul 17 20:35:17.855711 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.855558    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" event={"ID":"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de","Type":"ContainerStarted","Data":"4a5100cd23fa77ddb9ff759040650e9bd600f53411e61c316dc413779df262f5"}
Jul 17 20:35:17.876182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:17.876127    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" podStartSLOduration=0.876111196 podStartE2EDuration="876.111196ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:35:17.875038272 +0000 UTC m=+2352.569014417" watchObservedRunningTime="2026-07-17 20:35:17.876111196 +0000 UTC m=+2352.570087342"
Jul 17 20:35:18.860888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:18.860849    2572 generic.go:356] "Generic (PLEG): container finished" podID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerID="62b378384d1533ecf19cb38c15b02a6710195c55734fe569d0e80598eebcfea6" exitCode=6
Jul 17 20:35:18.861284 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:18.860927    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" event={"ID":"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de","Type":"ContainerDied","Data":"62b378384d1533ecf19cb38c15b02a6710195c55734fe569d0e80598eebcfea6"}
Jul 17 20:35:18.861284 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:18.861124    2572 scope.go:117] "RemoveContainer" containerID="62b378384d1533ecf19cb38c15b02a6710195c55734fe569d0e80598eebcfea6"
Jul 17 20:35:19.866940 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:19.866908    2572 generic.go:356] "Generic (PLEG): container finished" podID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerID="ad898548070d7534571bcdd2a1017ac0a200d877d0f3038248428f44a4b1726a" exitCode=6
Jul 17 20:35:19.867424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:19.866972    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" event={"ID":"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de","Type":"ContainerDied","Data":"ad898548070d7534571bcdd2a1017ac0a200d877d0f3038248428f44a4b1726a"}
Jul 17 20:35:19.867424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:19.867011    2572 scope.go:117] "RemoveContainer" containerID="62b378384d1533ecf19cb38c15b02a6710195c55734fe569d0e80598eebcfea6"
Jul 17 20:35:19.867424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:19.867249    2572 scope.go:117] "RemoveContainer" containerID="ad898548070d7534571bcdd2a1017ac0a200d877d0f3038248428f44a4b1726a"
Jul 17 20:35:19.867563 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:19.867453    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rs57h_odh-ai-gateway-infra(c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de"
Jul 17 20:35:20.873646 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:20.873621    2572 scope.go:117] "RemoveContainer" containerID="ad898548070d7534571bcdd2a1017ac0a200d877d0f3038248428f44a4b1726a"
Jul 17 20:35:20.874015 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:20.873806    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rs57h_odh-ai-gateway-infra(c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de"
Jul 17 20:35:27.848421 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:27.848387    2572 scope.go:117] "RemoveContainer" containerID="e65e3a946236feed976372282a030bb8ecd860048525c5962b89446f120c0091"
Jul 17 20:35:27.848800 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:27.848702    2572 scope.go:117] "RemoveContainer" containerID="c69b23c237b5f4bbcec873ad9ee38d21424f7c7d09b8b54d94e96db2e86b1c10"
Jul 17 20:35:28.911755 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.911715    2572 generic.go:356] "Generic (PLEG): container finished" podID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerID="449096006af22328393ec4b0d8ff3611a30d0588451317d85357bda6b47cf5bc" exitCode=6
Jul 17 20:35:28.912221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.911802    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" event={"ID":"0a263319-6bb0-4ad1-a589-36e2e612c8a7","Type":"ContainerDied","Data":"449096006af22328393ec4b0d8ff3611a30d0588451317d85357bda6b47cf5bc"}
Jul 17 20:35:28.912221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.911870    2572 scope.go:117] "RemoveContainer" containerID="e65e3a946236feed976372282a030bb8ecd860048525c5962b89446f120c0091"
Jul 17 20:35:28.912221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.912192    2572 scope.go:117] "RemoveContainer" containerID="449096006af22328393ec4b0d8ff3611a30d0588451317d85357bda6b47cf5bc"
Jul 17 20:35:28.912431 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:28.912412    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh_odh-ai-gateway-infra(0a263319-6bb0-4ad1-a589-36e2e612c8a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7"
Jul 17 20:35:28.913685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.913666    2572 generic.go:356] "Generic (PLEG): container finished" podID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerID="08d250964e0199aee9203fb6c03d1226a13fda2e12780764f02527673c687081" exitCode=6
Jul 17 20:35:28.913781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.913699    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" event={"ID":"ac30027b-4ab8-433b-ab43-8e3768cf0baf","Type":"ContainerDied","Data":"08d250964e0199aee9203fb6c03d1226a13fda2e12780764f02527673c687081"}
Jul 17 20:35:28.913968 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.913955    2572 scope.go:117] "RemoveContainer" containerID="08d250964e0199aee9203fb6c03d1226a13fda2e12780764f02527673c687081"
Jul 17 20:35:28.914126 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:28.914111    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-6jcfr_odh-ai-gateway-infra(ac30027b-4ab8-433b-ab43-8e3768cf0baf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf"
Jul 17 20:35:28.922124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:28.922107    2572 scope.go:117] "RemoveContainer" containerID="c69b23c237b5f4bbcec873ad9ee38d21424f7c7d09b8b54d94e96db2e86b1c10"
Jul 17 20:35:29.969688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:29.969651    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"]
Jul 17 20:35:29.998296 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:29.998258    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"]
Jul 17 20:35:30.110183 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.110156    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"
Jul 17 20:35:30.148378 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.148351    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"
Jul 17 20:35:30.221002 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.220913    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a263319-6bb0-4ad1-a589-36e2e612c8a7-maas-api-service-ca\") pod \"0a263319-6bb0-4ad1-a589-36e2e612c8a7\" (UID: \"0a263319-6bb0-4ad1-a589-36e2e612c8a7\") "
Jul 17 20:35:30.221002 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.220981    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac30027b-4ab8-433b-ab43-8e3768cf0baf-maas-api-service-ca\") pod \"ac30027b-4ab8-433b-ab43-8e3768cf0baf\" (UID: \"ac30027b-4ab8-433b-ab43-8e3768cf0baf\") "
Jul 17 20:35:30.221333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.221312    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac30027b-4ab8-433b-ab43-8e3768cf0baf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac30027b-4ab8-433b-ab43-8e3768cf0baf" (UID: "ac30027b-4ab8-433b-ab43-8e3768cf0baf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:35:30.221378 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.221309    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a263319-6bb0-4ad1-a589-36e2e612c8a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a263319-6bb0-4ad1-a589-36e2e612c8a7" (UID: "0a263319-6bb0-4ad1-a589-36e2e612c8a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:35:30.322420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.322385    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a263319-6bb0-4ad1-a589-36e2e612c8a7-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:35:30.322420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.322414    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac30027b-4ab8-433b-ab43-8e3768cf0baf-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:35:30.926206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.926174    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"
Jul 17 20:35:30.926206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.926189    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh" event={"ID":"0a263319-6bb0-4ad1-a589-36e2e612c8a7","Type":"ContainerDied","Data":"2289a27f02e37f95dbe5e29b51eb4c7a8e18dfe75ec059479b6e40c45c0606b0"}
Jul 17 20:35:30.926463 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.926253    2572 scope.go:117] "RemoveContainer" containerID="449096006af22328393ec4b0d8ff3611a30d0588451317d85357bda6b47cf5bc"
Jul 17 20:35:30.927556 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.927536    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr" event={"ID":"ac30027b-4ab8-433b-ab43-8e3768cf0baf","Type":"ContainerDied","Data":"e0fe274ca5a5c4a5a05b6782339eb40e8e5326ec4dfb97c0bd8e39086bcd5c81"}
Jul 17 20:35:30.927675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.927574    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"
Jul 17 20:35:30.934153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.934132    2572 scope.go:117] "RemoveContainer" containerID="08d250964e0199aee9203fb6c03d1226a13fda2e12780764f02527673c687081"
Jul 17 20:35:30.952475 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.952450    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"]
Jul 17 20:35:30.958689 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.958669    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-p6fbh"]
Jul 17 20:35:30.969528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.969508    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"]
Jul 17 20:35:30.974320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:30.974298    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-6jcfr"]
Jul 17 20:35:31.853084 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:31.853034    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" path="/var/lib/kubelet/pods/0a263319-6bb0-4ad1-a589-36e2e612c8a7/volumes"
Jul 17 20:35:31.853390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:31.853378    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" path="/var/lib/kubelet/pods/ac30027b-4ab8-433b-ab43-8e3768cf0baf/volumes"
Jul 17 20:35:35.852348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:35.851801    2572 scope.go:117] "RemoveContainer" containerID="ad898548070d7534571bcdd2a1017ac0a200d877d0f3038248428f44a4b1726a"
Jul 17 20:35:36.957218 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:36.957140    2572 generic.go:356] "Generic (PLEG): container finished" podID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerID="3193d9ce1bf1ea1ca783e30638ffba2121fb3d097034584376282230d57eb8cf" exitCode=6
Jul 17 20:35:36.957685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:36.957216    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" event={"ID":"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de","Type":"ContainerDied","Data":"3193d9ce1bf1ea1ca783e30638ffba2121fb3d097034584376282230d57eb8cf"}
Jul 17 20:35:36.957685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:36.957281    2572 scope.go:117] "RemoveContainer" containerID="ad898548070d7534571bcdd2a1017ac0a200d877d0f3038248428f44a4b1726a"
Jul 17 20:35:36.957685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:36.957635    2572 scope.go:117] "RemoveContainer" containerID="3193d9ce1bf1ea1ca783e30638ffba2121fb3d097034584376282230d57eb8cf"
Jul 17 20:35:36.957869 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:36.957849    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rs57h_odh-ai-gateway-infra(c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de"
Jul 17 20:35:37.990476 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:37.990442    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"]
Jul 17 20:35:38.126278 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.126255    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"
Jul 17 20:35:38.178359 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.178326    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de-maas-api-service-ca\") pod \"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de\" (UID: \"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de\") "
Jul 17 20:35:38.178712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.178687    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" (UID: "c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:35:38.279429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.279341    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:35:38.968662 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.968635    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"
Jul 17 20:35:38.968662 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.968651    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h" event={"ID":"c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de","Type":"ContainerDied","Data":"4a5100cd23fa77ddb9ff759040650e9bd600f53411e61c316dc413779df262f5"}
Jul 17 20:35:38.968876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.968717    2572 scope.go:117] "RemoveContainer" containerID="3193d9ce1bf1ea1ca783e30638ffba2121fb3d097034584376282230d57eb8cf"
Jul 17 20:35:38.994702 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:38.994671    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"]
Jul 17 20:35:39.000687 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:39.000666    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rs57h"]
Jul 17 20:35:39.853379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:39.853338    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" path="/var/lib/kubelet/pods/c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de/volumes"
Jul 17 20:35:45.031231 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031188    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"]
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031610    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031628    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031642    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031650    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031661    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031669    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031685    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031693    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:35:45.031705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031702    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031712    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031725    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031733    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031742    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031750    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031767    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031775    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031854    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031865    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031876    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031886    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a69bf7e-c56e-456b-8bcd-56f7e2d2e93e" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031895    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031903    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031911    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:35:45.032189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.031920    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:35:45.034992 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.034972    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"
Jul 17 20:35:45.037774 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.037748    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:35:45.037876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.037759    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:35:45.042708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.042681    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"]
Jul 17 20:35:45.130737 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.130700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2e273-c8ce-4d00-8d20-1df2887f4bea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv\" (UID: \"d9c2e273-c8ce-4d00-8d20-1df2887f4bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"
Jul 17 20:35:45.232137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.232099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2e273-c8ce-4d00-8d20-1df2887f4bea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv\" (UID: \"d9c2e273-c8ce-4d00-8d20-1df2887f4bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"
Jul 17 20:35:45.232678 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.232657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2e273-c8ce-4d00-8d20-1df2887f4bea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv\" (UID: \"d9c2e273-c8ce-4d00-8d20-1df2887f4bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"
Jul 17 20:35:45.345454 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.345371    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"
Jul 17 20:35:45.473624 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.473592    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"]
Jul 17 20:35:45.475634 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:35:45.475610    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c2e273_c8ce_4d00_8d20_1df2887f4bea.slice/crio-ea345b0af69826fde7b1513497a9120708f145629a2dcc39f46e0dfb3c9af94c WatchSource:0}: Error finding container ea345b0af69826fde7b1513497a9120708f145629a2dcc39f46e0dfb3c9af94c: Status 404 returned error can't find the container with id ea345b0af69826fde7b1513497a9120708f145629a2dcc39f46e0dfb3c9af94c
Jul 17 20:35:45.999782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.999749    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" event={"ID":"d9c2e273-c8ce-4d00-8d20-1df2887f4bea","Type":"ContainerStarted","Data":"bd9065771de67ab6ad062448b5d11972504186bda5ef96f6de2f2534b77fd7ba"}
Jul 17 20:35:45.999959 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:45.999801    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" event={"ID":"d9c2e273-c8ce-4d00-8d20-1df2887f4bea","Type":"ContainerStarted","Data":"ea345b0af69826fde7b1513497a9120708f145629a2dcc39f46e0dfb3c9af94c"}
Jul 17 20:35:46.027884 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:46.027831    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" podStartSLOduration=1.027815165 podStartE2EDuration="1.027815165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:35:46.027497353 +0000 UTC m=+2380.721473500" watchObservedRunningTime="2026-07-17 20:35:46.027815165 +0000 UTC m=+2380.721791310"
Jul 17 20:35:47.005805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:47.005772    2572 generic.go:356] "Generic (PLEG): container finished" podID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerID="bd9065771de67ab6ad062448b5d11972504186bda5ef96f6de2f2534b77fd7ba" exitCode=7
Jul 17 20:35:47.006272 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:47.005850    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" event={"ID":"d9c2e273-c8ce-4d00-8d20-1df2887f4bea","Type":"ContainerDied","Data":"bd9065771de67ab6ad062448b5d11972504186bda5ef96f6de2f2534b77fd7ba"}
Jul 17 20:35:47.006272 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:47.006083    2572 scope.go:117] "RemoveContainer" containerID="bd9065771de67ab6ad062448b5d11972504186bda5ef96f6de2f2534b77fd7ba"
Jul 17 20:35:48.012019 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:48.011984    2572 generic.go:356] "Generic (PLEG): container finished" podID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerID="a2e55f658422e137c735f4247f643e25f0416b8767e7c67e4ca63849ddf3fb88" exitCode=7
Jul 17 20:35:48.012508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:48.012076    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" event={"ID":"d9c2e273-c8ce-4d00-8d20-1df2887f4bea","Type":"ContainerDied","Data":"a2e55f658422e137c735f4247f643e25f0416b8767e7c67e4ca63849ddf3fb88"}
Jul 17 20:35:48.012508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:48.012122    2572 scope.go:117] "RemoveContainer" containerID="bd9065771de67ab6ad062448b5d11972504186bda5ef96f6de2f2534b77fd7ba"
Jul 17 20:35:48.012508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:48.012315    2572 scope.go:117] "RemoveContainer" containerID="a2e55f658422e137c735f4247f643e25f0416b8767e7c67e4ca63849ddf3fb88"
Jul 17 20:35:48.012672 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:48.012557    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv_odh-ai-gateway-infra(d9c2e273-c8ce-4d00-8d20-1df2887f4bea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea"
Jul 17 20:35:49.018297 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:35:49.018269    2572 scope.go:117] "RemoveContainer" containerID="a2e55f658422e137c735f4247f643e25f0416b8767e7c67e4ca63849ddf3fb88"
Jul 17 20:35:49.018688 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:35:49.018455    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv_odh-ai-gateway-infra(d9c2e273-c8ce-4d00-8d20-1df2887f4bea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea"
Jul 17 20:36:00.848417 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:00.848388    2572 scope.go:117] "RemoveContainer" containerID="a2e55f658422e137c735f4247f643e25f0416b8767e7c67e4ca63849ddf3fb88"
Jul 17 20:36:01.070305 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:01.070276    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" event={"ID":"d9c2e273-c8ce-4d00-8d20-1df2887f4bea","Type":"ContainerStarted","Data":"accc614ee60f1b80d044fd224fa74ed7fa7a142bd3fb2d97e62e5da5024ffd04"}
Jul 17 20:36:02.076401 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:02.076365    2572 generic.go:356] "Generic (PLEG): container finished" podID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerID="accc614ee60f1b80d044fd224fa74ed7fa7a142bd3fb2d97e62e5da5024ffd04" exitCode=7
Jul 17 20:36:02.076808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:02.076398    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" event={"ID":"d9c2e273-c8ce-4d00-8d20-1df2887f4bea","Type":"ContainerDied","Data":"accc614ee60f1b80d044fd224fa74ed7fa7a142bd3fb2d97e62e5da5024ffd04"}
Jul 17 20:36:02.076808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:02.076458    2572 scope.go:117] "RemoveContainer" containerID="a2e55f658422e137c735f4247f643e25f0416b8767e7c67e4ca63849ddf3fb88"
Jul 17 20:36:02.076808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:02.076694    2572 scope.go:117] "RemoveContainer" containerID="accc614ee60f1b80d044fd224fa74ed7fa7a142bd3fb2d97e62e5da5024ffd04"
Jul 17 20:36:02.076922 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:02.076879    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv_odh-ai-gateway-infra(d9c2e273-c8ce-4d00-8d20-1df2887f4bea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea"
Jul 17 20:36:02.111625 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:02.111596    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"]
Jul 17 20:36:03.215832 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:03.215807    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"
Jul 17 20:36:03.283291 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:03.283262    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2e273-c8ce-4d00-8d20-1df2887f4bea-maas-api-service-ca\") pod \"d9c2e273-c8ce-4d00-8d20-1df2887f4bea\" (UID: \"d9c2e273-c8ce-4d00-8d20-1df2887f4bea\") "
Jul 17 20:36:03.283611 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:03.283589    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2e273-c8ce-4d00-8d20-1df2887f4bea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9c2e273-c8ce-4d00-8d20-1df2887f4bea" (UID: "d9c2e273-c8ce-4d00-8d20-1df2887f4bea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:36:03.384065 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:03.383969    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2e273-c8ce-4d00-8d20-1df2887f4bea-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:36:04.087991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:04.087965    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"
Jul 17 20:36:04.087991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:04.087974    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv" event={"ID":"d9c2e273-c8ce-4d00-8d20-1df2887f4bea","Type":"ContainerDied","Data":"ea345b0af69826fde7b1513497a9120708f145629a2dcc39f46e0dfb3c9af94c"}
Jul 17 20:36:04.088221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:04.088031    2572 scope.go:117] "RemoveContainer" containerID="accc614ee60f1b80d044fd224fa74ed7fa7a142bd3fb2d97e62e5da5024ffd04"
Jul 17 20:36:04.109600 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:04.109571    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"]
Jul 17 20:36:04.112277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:04.112226    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qwnjv"]
Jul 17 20:36:05.853469 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:05.853438    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" path="/var/lib/kubelet/pods/d9c2e273-c8ce-4d00-8d20-1df2887f4bea/volumes"
Jul 17 20:36:09.129999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.129965    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"]
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130282    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130294    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130307    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130313    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130329    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130335    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1bc33ca-bde0-44bb-9ea1-dc197c5dc3de" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130344    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130349    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130392    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a263319-6bb0-4ad1-a589-36e2e612c8a7" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130399    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130405    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac30027b-4ab8-433b-ab43-8e3768cf0baf" containerName="revoke-keys"
Jul 17 20:36:09.130459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.130410    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:09.134941 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.134923    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"
Jul 17 20:36:09.138848 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.138824    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:36:09.138964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.138854    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:36:09.144875 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.144850    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"]
Jul 17 20:36:09.228678 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.228645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae32d-ba07-4339-bb1d-1966fff83709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt\" (UID: \"d5eae32d-ba07-4339-bb1d-1966fff83709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"
Jul 17 20:36:09.329595 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.329558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae32d-ba07-4339-bb1d-1966fff83709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt\" (UID: \"d5eae32d-ba07-4339-bb1d-1966fff83709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"
Jul 17 20:36:09.330203 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.330185    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae32d-ba07-4339-bb1d-1966fff83709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt\" (UID: \"d5eae32d-ba07-4339-bb1d-1966fff83709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"
Jul 17 20:36:09.446749 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.446652    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"
Jul 17 20:36:09.581402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:09.581379    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"]
Jul 17 20:36:09.583428 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:36:09.583398    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eae32d_ba07_4339_bb1d_1966fff83709.slice/crio-678718c6070a212d725d30e81000be37c213a4d2227d1387505f43f680d6261d WatchSource:0}: Error finding container 678718c6070a212d725d30e81000be37c213a4d2227d1387505f43f680d6261d: Status 404 returned error can't find the container with id 678718c6070a212d725d30e81000be37c213a4d2227d1387505f43f680d6261d
Jul 17 20:36:10.113523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:10.113434    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" event={"ID":"d5eae32d-ba07-4339-bb1d-1966fff83709","Type":"ContainerStarted","Data":"0527bde433167042f766d74c1687b6624a98e2968d22b2796fd3f30b43113923"}
Jul 17 20:36:10.113523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:10.113495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" event={"ID":"d5eae32d-ba07-4339-bb1d-1966fff83709","Type":"ContainerStarted","Data":"678718c6070a212d725d30e81000be37c213a4d2227d1387505f43f680d6261d"}
Jul 17 20:36:10.144867 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:10.144823    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" podStartSLOduration=1.144809098 podStartE2EDuration="1.144809098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:36:10.14256725 +0000 UTC m=+2404.836543418" watchObservedRunningTime="2026-07-17 20:36:10.144809098 +0000 UTC m=+2404.838785244"
Jul 17 20:36:11.061636 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.061603    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"]
Jul 17 20:36:11.061942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.061930    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:11.062008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.061943    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:11.062008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.061998    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c2e273-c8ce-4d00-8d20-1df2887f4bea" containerName="revoke-keys"
Jul 17 20:36:11.064893 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.064879    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"
Jul 17 20:36:11.073612 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.073585    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"]
Jul 17 20:36:11.119134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.119098    2572 generic.go:356] "Generic (PLEG): container finished" podID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerID="0527bde433167042f766d74c1687b6624a98e2968d22b2796fd3f30b43113923" exitCode=6
Jul 17 20:36:11.119319 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.119153    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" event={"ID":"d5eae32d-ba07-4339-bb1d-1966fff83709","Type":"ContainerDied","Data":"0527bde433167042f766d74c1687b6624a98e2968d22b2796fd3f30b43113923"}
Jul 17 20:36:11.119384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.119352    2572 scope.go:117] "RemoveContainer" containerID="0527bde433167042f766d74c1687b6624a98e2968d22b2796fd3f30b43113923"
Jul 17 20:36:11.144397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.144372    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c492c-97d6-413f-aae8-7c471189c656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-7t5jt\" (UID: \"c72c492c-97d6-413f-aae8-7c471189c656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"
Jul 17 20:36:11.244740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.244704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c492c-97d6-413f-aae8-7c471189c656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-7t5jt\" (UID: \"c72c492c-97d6-413f-aae8-7c471189c656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"
Jul 17 20:36:11.245338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.245316    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c492c-97d6-413f-aae8-7c471189c656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-7t5jt\" (UID: \"c72c492c-97d6-413f-aae8-7c471189c656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"
Jul 17 20:36:11.375749 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.375717    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"
Jul 17 20:36:11.524344 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:11.524317    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"]
Jul 17 20:36:11.525818 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:36:11.525788    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72c492c_97d6_413f_aae8_7c471189c656.slice/crio-ab245bda1192943edafbbb117fdf779b4bc3224ccb57e0217a075766fd92e60e WatchSource:0}: Error finding container ab245bda1192943edafbbb117fdf779b4bc3224ccb57e0217a075766fd92e60e: Status 404 returned error can't find the container with id ab245bda1192943edafbbb117fdf779b4bc3224ccb57e0217a075766fd92e60e
Jul 17 20:36:12.126027 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:12.125942    2572 generic.go:356] "Generic (PLEG): container finished" podID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerID="2fe7c50ebeeb118c7f9c290ae1b45891a1af98aede4def3776f3c152d00c292b" exitCode=6
Jul 17 20:36:12.126216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:12.126031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" event={"ID":"d5eae32d-ba07-4339-bb1d-1966fff83709","Type":"ContainerDied","Data":"2fe7c50ebeeb118c7f9c290ae1b45891a1af98aede4def3776f3c152d00c292b"}
Jul 17 20:36:12.126216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:12.126131    2572 scope.go:117] "RemoveContainer" containerID="0527bde433167042f766d74c1687b6624a98e2968d22b2796fd3f30b43113923"
Jul 17 20:36:12.126374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:12.126358    2572 scope.go:117] "RemoveContainer" containerID="2fe7c50ebeeb118c7f9c290ae1b45891a1af98aede4def3776f3c152d00c292b"
Jul 17 20:36:12.126615 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:12.126594    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt_odh-ai-gateway-infra(d5eae32d-ba07-4339-bb1d-1966fff83709)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709"
Jul 17 20:36:12.127712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:12.127681    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" event={"ID":"c72c492c-97d6-413f-aae8-7c471189c656","Type":"ContainerStarted","Data":"aac72694783dd5c2804ab47373bb0f636963f88cbd0bb7cf456247debd9c0215"}
Jul 17 20:36:12.127809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:12.127720    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" event={"ID":"c72c492c-97d6-413f-aae8-7c471189c656","Type":"ContainerStarted","Data":"ab245bda1192943edafbbb117fdf779b4bc3224ccb57e0217a075766fd92e60e"}
Jul 17 20:36:12.170793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:12.170748    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" podStartSLOduration=1.1707335429999999 podStartE2EDuration="1.170733543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:36:12.170704936 +0000 UTC m=+2406.864681083" watchObservedRunningTime="2026-07-17 20:36:12.170733543 +0000 UTC m=+2406.864709818"
Jul 17 20:36:13.134628 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:13.134597    2572 scope.go:117] "RemoveContainer" containerID="2fe7c50ebeeb118c7f9c290ae1b45891a1af98aede4def3776f3c152d00c292b"
Jul 17 20:36:13.135075 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:13.134796    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt_odh-ai-gateway-infra(d5eae32d-ba07-4339-bb1d-1966fff83709)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709"
Jul 17 20:36:13.135747 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:13.135723    2572 generic.go:356] "Generic (PLEG): container finished" podID="c72c492c-97d6-413f-aae8-7c471189c656" containerID="aac72694783dd5c2804ab47373bb0f636963f88cbd0bb7cf456247debd9c0215" exitCode=6
Jul 17 20:36:13.135830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:13.135805    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" event={"ID":"c72c492c-97d6-413f-aae8-7c471189c656","Type":"ContainerDied","Data":"aac72694783dd5c2804ab47373bb0f636963f88cbd0bb7cf456247debd9c0215"}
Jul 17 20:36:13.135938 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:13.135926    2572 scope.go:117] "RemoveContainer" containerID="aac72694783dd5c2804ab47373bb0f636963f88cbd0bb7cf456247debd9c0215"
Jul 17 20:36:14.142005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:14.141978    2572 generic.go:356] "Generic (PLEG): container finished" podID="c72c492c-97d6-413f-aae8-7c471189c656" containerID="8921902c273e03298c348695273782060736e08b6c0039d7558dad491b4e7087" exitCode=6
Jul 17 20:36:14.142399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:14.142026    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" event={"ID":"c72c492c-97d6-413f-aae8-7c471189c656","Type":"ContainerDied","Data":"8921902c273e03298c348695273782060736e08b6c0039d7558dad491b4e7087"}
Jul 17 20:36:14.142399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:14.142097    2572 scope.go:117] "RemoveContainer" containerID="aac72694783dd5c2804ab47373bb0f636963f88cbd0bb7cf456247debd9c0215"
Jul 17 20:36:14.142399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:14.142265    2572 scope.go:117] "RemoveContainer" containerID="8921902c273e03298c348695273782060736e08b6c0039d7558dad491b4e7087"
Jul 17 20:36:14.142526 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:14.142462    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-7t5jt_odh-ai-gateway-infra(c72c492c-97d6-413f-aae8-7c471189c656)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" podUID="c72c492c-97d6-413f-aae8-7c471189c656"
Jul 17 20:36:15.148800 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:15.148776    2572 scope.go:117] "RemoveContainer" containerID="8921902c273e03298c348695273782060736e08b6c0039d7558dad491b4e7087"
Jul 17 20:36:15.149178 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:15.148941    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-7t5jt_odh-ai-gateway-infra(c72c492c-97d6-413f-aae8-7c471189c656)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" podUID="c72c492c-97d6-413f-aae8-7c471189c656"
Jul 17 20:36:17.275679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.275637    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"]
Jul 17 20:36:17.280261 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.280238    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"
Jul 17 20:36:17.293259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.293235    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"]
Jul 17 20:36:17.395262 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.395230    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5066273-73e6-4b3e-90ed-8ec3134720fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-t52tc\" (UID: \"f5066273-73e6-4b3e-90ed-8ec3134720fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"
Jul 17 20:36:17.496147 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.496113    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5066273-73e6-4b3e-90ed-8ec3134720fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-t52tc\" (UID: \"f5066273-73e6-4b3e-90ed-8ec3134720fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"
Jul 17 20:36:17.496721 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.496701    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5066273-73e6-4b3e-90ed-8ec3134720fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-t52tc\" (UID: \"f5066273-73e6-4b3e-90ed-8ec3134720fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"
Jul 17 20:36:17.589280 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.589191    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"
Jul 17 20:36:17.724739 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:17.724716    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"]
Jul 17 20:36:17.726553 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:36:17.726520    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5066273_73e6_4b3e_90ed_8ec3134720fa.slice/crio-5cb540e77ba8360ccaf9b6149f08548266777f9cceefb3a0ee4ddf00bf9f9b92 WatchSource:0}: Error finding container 5cb540e77ba8360ccaf9b6149f08548266777f9cceefb3a0ee4ddf00bf9f9b92: Status 404 returned error can't find the container with id 5cb540e77ba8360ccaf9b6149f08548266777f9cceefb3a0ee4ddf00bf9f9b92
Jul 17 20:36:18.162012 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:18.161980    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" event={"ID":"f5066273-73e6-4b3e-90ed-8ec3134720fa","Type":"ContainerStarted","Data":"23fd1fbb115e981d9e6625cd80ff685a7cbd054b57cbf301593e009feabfa398"}
Jul 17 20:36:18.162184 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:18.162029    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" event={"ID":"f5066273-73e6-4b3e-90ed-8ec3134720fa","Type":"ContainerStarted","Data":"5cb540e77ba8360ccaf9b6149f08548266777f9cceefb3a0ee4ddf00bf9f9b92"}
Jul 17 20:36:18.181339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:18.181296    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" podStartSLOduration=1.181284101 podStartE2EDuration="1.181284101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:36:18.179625143 +0000 UTC m=+2412.873601291" watchObservedRunningTime="2026-07-17 20:36:18.181284101 +0000 UTC m=+2412.875260247"
Jul 17 20:36:19.167377 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:19.167342    2572 generic.go:356] "Generic (PLEG): container finished" podID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerID="23fd1fbb115e981d9e6625cd80ff685a7cbd054b57cbf301593e009feabfa398" exitCode=6
Jul 17 20:36:19.167789 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:19.167438    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" event={"ID":"f5066273-73e6-4b3e-90ed-8ec3134720fa","Type":"ContainerDied","Data":"23fd1fbb115e981d9e6625cd80ff685a7cbd054b57cbf301593e009feabfa398"}
Jul 17 20:36:19.167789 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:19.167603    2572 scope.go:117] "RemoveContainer" containerID="23fd1fbb115e981d9e6625cd80ff685a7cbd054b57cbf301593e009feabfa398"
Jul 17 20:36:20.173708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:20.173613    2572 generic.go:356] "Generic (PLEG): container finished" podID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerID="3424c678f26ad979c8c62f3137b3974efbe95c8b9f6aa13f70b0950d4fd38005" exitCode=6
Jul 17 20:36:20.173708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:20.173683    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" event={"ID":"f5066273-73e6-4b3e-90ed-8ec3134720fa","Type":"ContainerDied","Data":"3424c678f26ad979c8c62f3137b3974efbe95c8b9f6aa13f70b0950d4fd38005"}
Jul 17 20:36:20.174237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:20.173731    2572 scope.go:117] "RemoveContainer" containerID="23fd1fbb115e981d9e6625cd80ff685a7cbd054b57cbf301593e009feabfa398"
Jul 17 20:36:20.174237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:20.173912    2572 scope.go:117] "RemoveContainer" containerID="3424c678f26ad979c8c62f3137b3974efbe95c8b9f6aa13f70b0950d4fd38005"
Jul 17 20:36:20.174237 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:20.174163    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-t52tc_odh-ai-gateway-infra(f5066273-73e6-4b3e-90ed-8ec3134720fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa"
Jul 17 20:36:21.179405 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:21.179377    2572 scope.go:117] "RemoveContainer" containerID="3424c678f26ad979c8c62f3137b3974efbe95c8b9f6aa13f70b0950d4fd38005"
Jul 17 20:36:21.179786 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:21.179540    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-t52tc_odh-ai-gateway-infra(f5066273-73e6-4b3e-90ed-8ec3134720fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa"
Jul 17 20:36:25.852246 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:25.852214    2572 scope.go:117] "RemoveContainer" containerID="2fe7c50ebeeb118c7f9c290ae1b45891a1af98aede4def3776f3c152d00c292b"
Jul 17 20:36:26.202380 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:26.202342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" event={"ID":"d5eae32d-ba07-4339-bb1d-1966fff83709","Type":"ContainerStarted","Data":"1177fb6ecdcb5d4bce7dfd82bd688771c424b1b75b669dfe7b5a048f0220ed67"}
Jul 17 20:36:27.209451 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:27.209419    2572 generic.go:356] "Generic (PLEG): container finished" podID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerID="1177fb6ecdcb5d4bce7dfd82bd688771c424b1b75b669dfe7b5a048f0220ed67" exitCode=6
Jul 17 20:36:27.209929 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:27.209501    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" event={"ID":"d5eae32d-ba07-4339-bb1d-1966fff83709","Type":"ContainerDied","Data":"1177fb6ecdcb5d4bce7dfd82bd688771c424b1b75b669dfe7b5a048f0220ed67"}
Jul 17 20:36:27.209929 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:27.209573    2572 scope.go:117] "RemoveContainer" containerID="2fe7c50ebeeb118c7f9c290ae1b45891a1af98aede4def3776f3c152d00c292b"
Jul 17 20:36:27.209929 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:27.209841    2572 scope.go:117] "RemoveContainer" containerID="1177fb6ecdcb5d4bce7dfd82bd688771c424b1b75b669dfe7b5a048f0220ed67"
Jul 17 20:36:27.210087 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:27.210036    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt_odh-ai-gateway-infra(d5eae32d-ba07-4339-bb1d-1966fff83709)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709"
Jul 17 20:36:27.237351 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:27.237319    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"]
Jul 17 20:36:28.343772 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:28.343750    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"
Jul 17 20:36:28.385508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:28.385478    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae32d-ba07-4339-bb1d-1966fff83709-maas-api-service-ca\") pod \"d5eae32d-ba07-4339-bb1d-1966fff83709\" (UID: \"d5eae32d-ba07-4339-bb1d-1966fff83709\") "
Jul 17 20:36:28.385831 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:28.385807    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5eae32d-ba07-4339-bb1d-1966fff83709-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5eae32d-ba07-4339-bb1d-1966fff83709" (UID: "d5eae32d-ba07-4339-bb1d-1966fff83709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:36:28.485980 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:28.485900    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eae32d-ba07-4339-bb1d-1966fff83709-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:36:29.220277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:29.220241    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt" event={"ID":"d5eae32d-ba07-4339-bb1d-1966fff83709","Type":"ContainerDied","Data":"678718c6070a212d725d30e81000be37c213a4d2227d1387505f43f680d6261d"}
Jul 17 20:36:29.220426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:29.220297    2572 scope.go:117] "RemoveContainer" containerID="1177fb6ecdcb5d4bce7dfd82bd688771c424b1b75b669dfe7b5a048f0220ed67"
Jul 17 20:36:29.220426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:29.220253    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"
Jul 17 20:36:29.248671 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:29.248643    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"]
Jul 17 20:36:29.255828 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:29.255809    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-n6mdt"]
Jul 17 20:36:29.848537 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:29.848504    2572 scope.go:117] "RemoveContainer" containerID="8921902c273e03298c348695273782060736e08b6c0039d7558dad491b4e7087"
Jul 17 20:36:29.853211 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:29.853188    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" path="/var/lib/kubelet/pods/d5eae32d-ba07-4339-bb1d-1966fff83709/volumes"
Jul 17 20:36:30.226911 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:30.226879    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" event={"ID":"c72c492c-97d6-413f-aae8-7c471189c656","Type":"ContainerStarted","Data":"420950aaa1e230faf0ba7dd1bc50a77f7017d835ea44be22eeaf8386547df6fe"}
Jul 17 20:36:31.238270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:31.238237    2572 generic.go:356] "Generic (PLEG): container finished" podID="c72c492c-97d6-413f-aae8-7c471189c656" containerID="420950aaa1e230faf0ba7dd1bc50a77f7017d835ea44be22eeaf8386547df6fe" exitCode=6
Jul 17 20:36:31.238753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:31.238291    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" event={"ID":"c72c492c-97d6-413f-aae8-7c471189c656","Type":"ContainerDied","Data":"420950aaa1e230faf0ba7dd1bc50a77f7017d835ea44be22eeaf8386547df6fe"}
Jul 17 20:36:31.238753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:31.238359    2572 scope.go:117] "RemoveContainer" containerID="8921902c273e03298c348695273782060736e08b6c0039d7558dad491b4e7087"
Jul 17 20:36:31.238753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:31.238640    2572 scope.go:117] "RemoveContainer" containerID="420950aaa1e230faf0ba7dd1bc50a77f7017d835ea44be22eeaf8386547df6fe"
Jul 17 20:36:31.238894 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:31.238834    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-7t5jt_odh-ai-gateway-infra(c72c492c-97d6-413f-aae8-7c471189c656)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" podUID="c72c492c-97d6-413f-aae8-7c471189c656"
Jul 17 20:36:31.253955 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:31.253934    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"]
Jul 17 20:36:32.370628 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:32.370604    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"
Jul 17 20:36:32.411745 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:32.411719    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c492c-97d6-413f-aae8-7c471189c656-maas-api-service-ca\") pod \"c72c492c-97d6-413f-aae8-7c471189c656\" (UID: \"c72c492c-97d6-413f-aae8-7c471189c656\") "
Jul 17 20:36:32.412090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:32.412065    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72c492c-97d6-413f-aae8-7c471189c656-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c72c492c-97d6-413f-aae8-7c471189c656" (UID: "c72c492c-97d6-413f-aae8-7c471189c656"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:36:32.512317 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:32.512239    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c72c492c-97d6-413f-aae8-7c471189c656-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:36:33.249518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:33.249487    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt" event={"ID":"c72c492c-97d6-413f-aae8-7c471189c656","Type":"ContainerDied","Data":"ab245bda1192943edafbbb117fdf779b4bc3224ccb57e0217a075766fd92e60e"}
Jul 17 20:36:33.249518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:33.249512    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"
Jul 17 20:36:33.249766 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:33.249543    2572 scope.go:117] "RemoveContainer" containerID="420950aaa1e230faf0ba7dd1bc50a77f7017d835ea44be22eeaf8386547df6fe"
Jul 17 20:36:33.271182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:33.271147    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"]
Jul 17 20:36:33.273502 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:33.273482    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-7t5jt"]
Jul 17 20:36:33.853025 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:33.852992    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c72c492c-97d6-413f-aae8-7c471189c656" path="/var/lib/kubelet/pods/c72c492c-97d6-413f-aae8-7c471189c656/volumes"
Jul 17 20:36:35.077333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077299    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"]
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077612    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077623    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077631    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077637    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077646    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077651    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077658    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077664    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077674    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077679    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077728    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077737    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077743    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:36:35.077844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.077749    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:36:35.082118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.082102    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"
Jul 17 20:36:35.087574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.087551    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"]
Jul 17 20:36:35.130743 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.130706    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625c38-e538-468f-a96a-3e8bc1ad0f58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2\" (UID: \"a0625c38-e538-468f-a96a-3e8bc1ad0f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"
Jul 17 20:36:35.231796 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.231756    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625c38-e538-468f-a96a-3e8bc1ad0f58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2\" (UID: \"a0625c38-e538-468f-a96a-3e8bc1ad0f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"
Jul 17 20:36:35.232385 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.232364    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625c38-e538-468f-a96a-3e8bc1ad0f58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2\" (UID: \"a0625c38-e538-468f-a96a-3e8bc1ad0f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"
Jul 17 20:36:35.392581 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.392497    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"
Jul 17 20:36:35.520852 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.520821    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"]
Jul 17 20:36:35.523925 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:36:35.523896    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0625c38_e538_468f_a96a_3e8bc1ad0f58.slice/crio-e5013d706f27d58f70602c7bb9f92f570c73a7d9d11661d916578bd20b9ceec8 WatchSource:0}: Error finding container e5013d706f27d58f70602c7bb9f92f570c73a7d9d11661d916578bd20b9ceec8: Status 404 returned error can't find the container with id e5013d706f27d58f70602c7bb9f92f570c73a7d9d11661d916578bd20b9ceec8
Jul 17 20:36:35.850248 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:35.850218    2572 scope.go:117] "RemoveContainer" containerID="3424c678f26ad979c8c62f3137b3974efbe95c8b9f6aa13f70b0950d4fd38005"
Jul 17 20:36:36.264207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:36.264175    2572 generic.go:356] "Generic (PLEG): container finished" podID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerID="abde9baa34a18375e55bc0d1a8839a3f0024bfcf35e47fa5efae751095da0047" exitCode=7
Jul 17 20:36:36.264685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:36.264216    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" event={"ID":"a0625c38-e538-468f-a96a-3e8bc1ad0f58","Type":"ContainerDied","Data":"abde9baa34a18375e55bc0d1a8839a3f0024bfcf35e47fa5efae751095da0047"}
Jul 17 20:36:36.264685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:36.264281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" event={"ID":"a0625c38-e538-468f-a96a-3e8bc1ad0f58","Type":"ContainerStarted","Data":"e5013d706f27d58f70602c7bb9f92f570c73a7d9d11661d916578bd20b9ceec8"}
Jul 17 20:36:36.264685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:36.264524    2572 scope.go:117] "RemoveContainer" containerID="abde9baa34a18375e55bc0d1a8839a3f0024bfcf35e47fa5efae751095da0047"
Jul 17 20:36:36.266082 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:36.266036    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" event={"ID":"f5066273-73e6-4b3e-90ed-8ec3134720fa","Type":"ContainerStarted","Data":"2bb5bbf5f366f2deac94bb79356d4f856abb9a1009ffeb662db4c61f1faf43d7"}
Jul 17 20:36:37.271608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.271578    2572 generic.go:356] "Generic (PLEG): container finished" podID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerID="ff1b3679271c36e51149980fd9d2e7e6a6dc3361d34c91a0e78e83e735ac3d45" exitCode=7
Jul 17 20:36:37.272058 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.271670    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" event={"ID":"a0625c38-e538-468f-a96a-3e8bc1ad0f58","Type":"ContainerDied","Data":"ff1b3679271c36e51149980fd9d2e7e6a6dc3361d34c91a0e78e83e735ac3d45"}
Jul 17 20:36:37.272058 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.271729    2572 scope.go:117] "RemoveContainer" containerID="abde9baa34a18375e55bc0d1a8839a3f0024bfcf35e47fa5efae751095da0047"
Jul 17 20:36:37.272058 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.271867    2572 scope.go:117] "RemoveContainer" containerID="ff1b3679271c36e51149980fd9d2e7e6a6dc3361d34c91a0e78e83e735ac3d45"
Jul 17 20:36:37.272181 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:37.272090    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2_odh-ai-gateway-infra(a0625c38-e538-468f-a96a-3e8bc1ad0f58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58"
Jul 17 20:36:37.273512 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.273492    2572 generic.go:356] "Generic (PLEG): container finished" podID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerID="2bb5bbf5f366f2deac94bb79356d4f856abb9a1009ffeb662db4c61f1faf43d7" exitCode=6
Jul 17 20:36:37.273583 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.273552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" event={"ID":"f5066273-73e6-4b3e-90ed-8ec3134720fa","Type":"ContainerDied","Data":"2bb5bbf5f366f2deac94bb79356d4f856abb9a1009ffeb662db4c61f1faf43d7"}
Jul 17 20:36:37.273778 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.273767    2572 scope.go:117] "RemoveContainer" containerID="2bb5bbf5f366f2deac94bb79356d4f856abb9a1009ffeb662db4c61f1faf43d7"
Jul 17 20:36:37.273914 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:37.273899    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-t52tc_odh-ai-gateway-infra(f5066273-73e6-4b3e-90ed-8ec3134720fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa"
Jul 17 20:36:37.281947 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.281931    2572 scope.go:117] "RemoveContainer" containerID="3424c678f26ad979c8c62f3137b3974efbe95c8b9f6aa13f70b0950d4fd38005"
Jul 17 20:36:37.310026 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:37.310000    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"]
Jul 17 20:36:38.279308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:38.279278    2572 scope.go:117] "RemoveContainer" containerID="ff1b3679271c36e51149980fd9d2e7e6a6dc3361d34c91a0e78e83e735ac3d45"
Jul 17 20:36:38.279794 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:38.279487    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2_odh-ai-gateway-infra(a0625c38-e538-468f-a96a-3e8bc1ad0f58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58"
Jul 17 20:36:38.420563 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:38.420536    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"
Jul 17 20:36:38.564389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:38.564303    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5066273-73e6-4b3e-90ed-8ec3134720fa-maas-api-service-ca\") pod \"f5066273-73e6-4b3e-90ed-8ec3134720fa\" (UID: \"f5066273-73e6-4b3e-90ed-8ec3134720fa\") "
Jul 17 20:36:38.564644 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:38.564622    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5066273-73e6-4b3e-90ed-8ec3134720fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5066273-73e6-4b3e-90ed-8ec3134720fa" (UID: "f5066273-73e6-4b3e-90ed-8ec3134720fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:36:38.664929 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:38.664894    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5066273-73e6-4b3e-90ed-8ec3134720fa-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:36:39.286174 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:39.286141    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc" event={"ID":"f5066273-73e6-4b3e-90ed-8ec3134720fa","Type":"ContainerDied","Data":"5cb540e77ba8360ccaf9b6149f08548266777f9cceefb3a0ee4ddf00bf9f9b92"}
Jul 17 20:36:39.286615 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:39.286196    2572 scope.go:117] "RemoveContainer" containerID="2bb5bbf5f366f2deac94bb79356d4f856abb9a1009ffeb662db4c61f1faf43d7"
Jul 17 20:36:39.286615 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:39.286157    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"
Jul 17 20:36:39.310325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:39.310301    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"]
Jul 17 20:36:39.312814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:39.312790    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-t52tc"]
Jul 17 20:36:39.852948 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:39.852918    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" path="/var/lib/kubelet/pods/f5066273-73e6-4b3e-90ed-8ec3134720fa/volumes"
Jul 17 20:36:52.848570 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:52.848538    2572 scope.go:117] "RemoveContainer" containerID="ff1b3679271c36e51149980fd9d2e7e6a6dc3361d34c91a0e78e83e735ac3d45"
Jul 17 20:36:53.348233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:53.348204    2572 generic.go:356] "Generic (PLEG): container finished" podID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerID="45a676780b4d43b3802426b1d4f1f40ba7c49fc98e24b5e0e454319cd488d0eb" exitCode=7
Jul 17 20:36:53.348367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:53.348274    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" event={"ID":"a0625c38-e538-468f-a96a-3e8bc1ad0f58","Type":"ContainerDied","Data":"45a676780b4d43b3802426b1d4f1f40ba7c49fc98e24b5e0e454319cd488d0eb"}
Jul 17 20:36:53.348367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:53.348335    2572 scope.go:117] "RemoveContainer" containerID="ff1b3679271c36e51149980fd9d2e7e6a6dc3361d34c91a0e78e83e735ac3d45"
Jul 17 20:36:53.348712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:53.348696    2572 scope.go:117] "RemoveContainer" containerID="45a676780b4d43b3802426b1d4f1f40ba7c49fc98e24b5e0e454319cd488d0eb"
Jul 17 20:36:53.348899 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:36:53.348880    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2_odh-ai-gateway-infra(a0625c38-e538-468f-a96a-3e8bc1ad0f58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58"
Jul 17 20:36:54.376973 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:54.376937    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"]
Jul 17 20:36:54.508155 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:54.508130    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"
Jul 17 20:36:54.584154 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:54.584115    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625c38-e538-468f-a96a-3e8bc1ad0f58-maas-api-service-ca\") pod \"a0625c38-e538-468f-a96a-3e8bc1ad0f58\" (UID: \"a0625c38-e538-468f-a96a-3e8bc1ad0f58\") "
Jul 17 20:36:54.584503 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:54.584480    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0625c38-e538-468f-a96a-3e8bc1ad0f58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0625c38-e538-468f-a96a-3e8bc1ad0f58" (UID: "a0625c38-e538-468f-a96a-3e8bc1ad0f58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:36:54.685212 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:54.685122    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625c38-e538-468f-a96a-3e8bc1ad0f58-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:36:55.360060 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:55.360011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2" event={"ID":"a0625c38-e538-468f-a96a-3e8bc1ad0f58","Type":"ContainerDied","Data":"e5013d706f27d58f70602c7bb9f92f570c73a7d9d11661d916578bd20b9ceec8"}
Jul 17 20:36:55.360251 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:55.360087    2572 scope.go:117] "RemoveContainer" containerID="45a676780b4d43b3802426b1d4f1f40ba7c49fc98e24b5e0e454319cd488d0eb"
Jul 17 20:36:55.360251 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:55.360023    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"
Jul 17 20:36:55.380774 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:55.380746    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"]
Jul 17 20:36:55.386977 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:55.386952    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-rxqd2"]
Jul 17 20:36:55.853829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:36:55.853795    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" path="/var/lib/kubelet/pods/a0625c38-e538-468f-a96a-3e8bc1ad0f58/volumes"
Jul 17 20:37:08.192183 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:08.192147    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:37:09.179142 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179108    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"]
Jul 17 20:37:09.179437 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179423    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:37:09.179494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179439    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:37:09.179494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179450    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179456    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179465    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:09.179494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179471    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:09.179494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179489    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179494    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179502    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179507    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179513    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179517    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179560    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179567    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179573    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179578    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179584    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5eae32d-ba07-4339-bb1d-1966fff83709" containerName="revoke-keys"
Jul 17 20:37:09.179709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.179591    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72c492c-97d6-413f-aae8-7c471189c656" containerName="revoke-keys"
Jul 17 20:37:09.182746 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.182727    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"
Jul 17 20:37:09.185205 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.185187    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:37:09.185315 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.185187    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:37:09.190537 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.190506    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"]
Jul 17 20:37:09.296996 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.296965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6a2197-2ddd-47d9-952c-aba449ba4d25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv\" (UID: \"5c6a2197-2ddd-47d9-952c-aba449ba4d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"
Jul 17 20:37:09.397830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.397801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6a2197-2ddd-47d9-952c-aba449ba4d25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv\" (UID: \"5c6a2197-2ddd-47d9-952c-aba449ba4d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"
Jul 17 20:37:09.398455 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.398436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6a2197-2ddd-47d9-952c-aba449ba4d25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv\" (UID: \"5c6a2197-2ddd-47d9-952c-aba449ba4d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"
Jul 17 20:37:09.493452 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.493422    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"
Jul 17 20:37:09.622083 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:09.622057    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"]
Jul 17 20:37:09.624628 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:37:09.624600    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6a2197_2ddd_47d9_952c_aba449ba4d25.slice/crio-2a0a6e7f9b973f89e649093abe0f18c6b720db9cf106ebff6e61eb763f6b4bbe WatchSource:0}: Error finding container 2a0a6e7f9b973f89e649093abe0f18c6b720db9cf106ebff6e61eb763f6b4bbe: Status 404 returned error can't find the container with id 2a0a6e7f9b973f89e649093abe0f18c6b720db9cf106ebff6e61eb763f6b4bbe
Jul 17 20:37:10.432784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:10.432754    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerID="73bc80011baeac4364e6aaab17c0c99e1a8c82f2987daa9e2680c5dc120c3e05" exitCode=6
Jul 17 20:37:10.433234 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:10.432844    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" event={"ID":"5c6a2197-2ddd-47d9-952c-aba449ba4d25","Type":"ContainerDied","Data":"73bc80011baeac4364e6aaab17c0c99e1a8c82f2987daa9e2680c5dc120c3e05"}
Jul 17 20:37:10.433234 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:10.432903    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" event={"ID":"5c6a2197-2ddd-47d9-952c-aba449ba4d25","Type":"ContainerStarted","Data":"2a0a6e7f9b973f89e649093abe0f18c6b720db9cf106ebff6e61eb763f6b4bbe"}
Jul 17 20:37:10.433234 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:10.433012    2572 scope.go:117] "RemoveContainer" containerID="73bc80011baeac4364e6aaab17c0c99e1a8c82f2987daa9e2680c5dc120c3e05"
Jul 17 20:37:11.103593 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.103563    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"]
Jul 17 20:37:11.103876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.103865    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:11.103922 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.103878    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:11.103962 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.103938    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5066273-73e6-4b3e-90ed-8ec3134720fa" containerName="revoke-keys"
Jul 17 20:37:11.103962 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.103945    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0625c38-e538-468f-a96a-3e8bc1ad0f58" containerName="revoke-keys"
Jul 17 20:37:11.106697 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.106679    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"
Jul 17 20:37:11.114857 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.114831    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"]
Jul 17 20:37:11.210645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.210616    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082ff1a9-ba00-4138-8881-4af88ad981a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-xz9tr\" (UID: \"082ff1a9-ba00-4138-8881-4af88ad981a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"
Jul 17 20:37:11.311492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.311396    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082ff1a9-ba00-4138-8881-4af88ad981a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-xz9tr\" (UID: \"082ff1a9-ba00-4138-8881-4af88ad981a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"
Jul 17 20:37:11.312020 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.312000    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082ff1a9-ba00-4138-8881-4af88ad981a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-xz9tr\" (UID: \"082ff1a9-ba00-4138-8881-4af88ad981a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"
Jul 17 20:37:11.417727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.417697    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"
Jul 17 20:37:11.440002 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.439951    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerID="649b5e38f2fca3bd80ecd603bfeb6e7e605503317f1297fff236dd8227080f0a" exitCode=6
Jul 17 20:37:11.440441 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.440072    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" event={"ID":"5c6a2197-2ddd-47d9-952c-aba449ba4d25","Type":"ContainerDied","Data":"649b5e38f2fca3bd80ecd603bfeb6e7e605503317f1297fff236dd8227080f0a"}
Jul 17 20:37:11.440441 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.440157    2572 scope.go:117] "RemoveContainer" containerID="73bc80011baeac4364e6aaab17c0c99e1a8c82f2987daa9e2680c5dc120c3e05"
Jul 17 20:37:11.440441 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.440403    2572 scope.go:117] "RemoveContainer" containerID="649b5e38f2fca3bd80ecd603bfeb6e7e605503317f1297fff236dd8227080f0a"
Jul 17 20:37:11.440660 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:11.440639    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv_odh-ai-gateway-infra(5c6a2197-2ddd-47d9-952c-aba449ba4d25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25"
Jul 17 20:37:11.550979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:11.550956    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"]
Jul 17 20:37:11.553205 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:37:11.553176    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082ff1a9_ba00_4138_8881_4af88ad981a4.slice/crio-97fe7b35b8c0dbfb0551b8a5a7aa10cb88161a8cf47290380c16e754691470e6 WatchSource:0}: Error finding container 97fe7b35b8c0dbfb0551b8a5a7aa10cb88161a8cf47290380c16e754691470e6: Status 404 returned error can't find the container with id 97fe7b35b8c0dbfb0551b8a5a7aa10cb88161a8cf47290380c16e754691470e6
Jul 17 20:37:12.446145 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:12.446042    2572 generic.go:356] "Generic (PLEG): container finished" podID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerID="6873e4bd40bed49256eeeae64036a7c2dd62f1472d0752b49ad9035548773696" exitCode=6
Jul 17 20:37:12.446145 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:12.446081    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" event={"ID":"082ff1a9-ba00-4138-8881-4af88ad981a4","Type":"ContainerDied","Data":"6873e4bd40bed49256eeeae64036a7c2dd62f1472d0752b49ad9035548773696"}
Jul 17 20:37:12.446590 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:12.446160    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" event={"ID":"082ff1a9-ba00-4138-8881-4af88ad981a4","Type":"ContainerStarted","Data":"97fe7b35b8c0dbfb0551b8a5a7aa10cb88161a8cf47290380c16e754691470e6"}
Jul 17 20:37:12.446590 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:12.446387    2572 scope.go:117] "RemoveContainer" containerID="6873e4bd40bed49256eeeae64036a7c2dd62f1472d0752b49ad9035548773696"
Jul 17 20:37:12.447921 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:12.447907    2572 scope.go:117] "RemoveContainer" containerID="649b5e38f2fca3bd80ecd603bfeb6e7e605503317f1297fff236dd8227080f0a"
Jul 17 20:37:12.448154 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:12.448039    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv_odh-ai-gateway-infra(5c6a2197-2ddd-47d9-952c-aba449ba4d25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25"
Jul 17 20:37:13.454981 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:13.454892    2572 generic.go:356] "Generic (PLEG): container finished" podID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerID="6c5add39d78dbe626665b11023076cbc3b91ff37624f96949250f0570211063d" exitCode=6
Jul 17 20:37:13.455453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:13.454981    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" event={"ID":"082ff1a9-ba00-4138-8881-4af88ad981a4","Type":"ContainerDied","Data":"6c5add39d78dbe626665b11023076cbc3b91ff37624f96949250f0570211063d"}
Jul 17 20:37:13.455453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:13.455039    2572 scope.go:117] "RemoveContainer" containerID="6873e4bd40bed49256eeeae64036a7c2dd62f1472d0752b49ad9035548773696"
Jul 17 20:37:13.455453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:13.455291    2572 scope.go:117] "RemoveContainer" containerID="6c5add39d78dbe626665b11023076cbc3b91ff37624f96949250f0570211063d"
Jul 17 20:37:13.455620 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:13.455493    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-xz9tr_odh-ai-gateway-infra(082ff1a9-ba00-4138-8881-4af88ad981a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4"
Jul 17 20:37:14.460986 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:14.460961    2572 scope.go:117] "RemoveContainer" containerID="6c5add39d78dbe626665b11023076cbc3b91ff37624f96949250f0570211063d"
Jul 17 20:37:14.461372 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:14.461200    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-xz9tr_odh-ai-gateway-infra(082ff1a9-ba00-4138-8881-4af88ad981a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4"
Jul 17 20:37:17.309194 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.309163    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"]
Jul 17 20:37:17.313319 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.313304    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"
Jul 17 20:37:17.320153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.319915    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"]
Jul 17 20:37:17.459195 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.459153    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546fa033-f3db-409f-83c5-171388682333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d\" (UID: \"546fa033-f3db-409f-83c5-171388682333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"
Jul 17 20:37:17.560452 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.560372    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546fa033-f3db-409f-83c5-171388682333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d\" (UID: \"546fa033-f3db-409f-83c5-171388682333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"
Jul 17 20:37:17.560958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.560940    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546fa033-f3db-409f-83c5-171388682333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d\" (UID: \"546fa033-f3db-409f-83c5-171388682333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"
Jul 17 20:37:17.624620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.624587    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"
Jul 17 20:37:17.755015 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:17.754989    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"]
Jul 17 20:37:17.757350 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:37:17.757313    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546fa033_f3db_409f_83c5_171388682333.slice/crio-ed04d0acc01641ed43c72018cfe9ac518cde7b6d5b9f300b34e8ab54a4176e11 WatchSource:0}: Error finding container ed04d0acc01641ed43c72018cfe9ac518cde7b6d5b9f300b34e8ab54a4176e11: Status 404 returned error can't find the container with id ed04d0acc01641ed43c72018cfe9ac518cde7b6d5b9f300b34e8ab54a4176e11
Jul 17 20:37:18.478085 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:18.478037    2572 generic.go:356] "Generic (PLEG): container finished" podID="546fa033-f3db-409f-83c5-171388682333" containerID="f8dc05c7ec9945d09ba1be035e93ea18d2a96e74c4efcfdc20ee5e9616c7a559" exitCode=6
Jul 17 20:37:18.478085 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:18.478079    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" event={"ID":"546fa033-f3db-409f-83c5-171388682333","Type":"ContainerDied","Data":"f8dc05c7ec9945d09ba1be035e93ea18d2a96e74c4efcfdc20ee5e9616c7a559"}
Jul 17 20:37:18.478483 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:18.478130    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" event={"ID":"546fa033-f3db-409f-83c5-171388682333","Type":"ContainerStarted","Data":"ed04d0acc01641ed43c72018cfe9ac518cde7b6d5b9f300b34e8ab54a4176e11"}
Jul 17 20:37:18.478483 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:18.478272    2572 scope.go:117] "RemoveContainer" containerID="f8dc05c7ec9945d09ba1be035e93ea18d2a96e74c4efcfdc20ee5e9616c7a559"
Jul 17 20:37:19.484569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:19.484531    2572 generic.go:356] "Generic (PLEG): container finished" podID="546fa033-f3db-409f-83c5-171388682333" containerID="7ca0ae830f76bcdb590e799d9c44d5f6dda27931d73f937b91ebae36d3d870cd" exitCode=6
Jul 17 20:37:19.484997 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:19.484617    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" event={"ID":"546fa033-f3db-409f-83c5-171388682333","Type":"ContainerDied","Data":"7ca0ae830f76bcdb590e799d9c44d5f6dda27931d73f937b91ebae36d3d870cd"}
Jul 17 20:37:19.484997 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:19.484704    2572 scope.go:117] "RemoveContainer" containerID="f8dc05c7ec9945d09ba1be035e93ea18d2a96e74c4efcfdc20ee5e9616c7a559"
Jul 17 20:37:19.484997 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:19.484799    2572 scope.go:117] "RemoveContainer" containerID="7ca0ae830f76bcdb590e799d9c44d5f6dda27931d73f937b91ebae36d3d870cd"
Jul 17 20:37:19.485170 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:19.484997    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d_odh-ai-gateway-infra(546fa033-f3db-409f-83c5-171388682333)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" podUID="546fa033-f3db-409f-83c5-171388682333"
Jul 17 20:37:20.490580 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:20.490555    2572 scope.go:117] "RemoveContainer" containerID="7ca0ae830f76bcdb590e799d9c44d5f6dda27931d73f937b91ebae36d3d870cd"
Jul 17 20:37:20.490976 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:20.490732    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d_odh-ai-gateway-infra(546fa033-f3db-409f-83c5-171388682333)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" podUID="546fa033-f3db-409f-83c5-171388682333"
Jul 17 20:37:23.848582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:23.848551    2572 scope.go:117] "RemoveContainer" containerID="649b5e38f2fca3bd80ecd603bfeb6e7e605503317f1297fff236dd8227080f0a"
Jul 17 20:37:24.507434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:24.507399    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerID="2e2810fd3b19b31a6aafd66e0f95b02990fb3ccdab5146bfb60d3cf5b6c14277" exitCode=6
Jul 17 20:37:24.507635 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:24.507473    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" event={"ID":"5c6a2197-2ddd-47d9-952c-aba449ba4d25","Type":"ContainerDied","Data":"2e2810fd3b19b31a6aafd66e0f95b02990fb3ccdab5146bfb60d3cf5b6c14277"}
Jul 17 20:37:24.507635 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:24.507532    2572 scope.go:117] "RemoveContainer" containerID="649b5e38f2fca3bd80ecd603bfeb6e7e605503317f1297fff236dd8227080f0a"
Jul 17 20:37:24.507884 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:24.507869    2572 scope.go:117] "RemoveContainer" containerID="2e2810fd3b19b31a6aafd66e0f95b02990fb3ccdab5146bfb60d3cf5b6c14277"
Jul 17 20:37:24.508122 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:24.508098    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv_odh-ai-gateway-infra(5c6a2197-2ddd-47d9-952c-aba449ba4d25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25"
Jul 17 20:37:25.534472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:25.534439    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"]
Jul 17 20:37:25.661434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:25.661413    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"
Jul 17 20:37:25.732425 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:25.732393    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6a2197-2ddd-47d9-952c-aba449ba4d25-maas-api-service-ca\") pod \"5c6a2197-2ddd-47d9-952c-aba449ba4d25\" (UID: \"5c6a2197-2ddd-47d9-952c-aba449ba4d25\") "
Jul 17 20:37:25.732763 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:25.732743    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6a2197-2ddd-47d9-952c-aba449ba4d25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c6a2197-2ddd-47d9-952c-aba449ba4d25" (UID: "5c6a2197-2ddd-47d9-952c-aba449ba4d25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:37:25.833888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:25.833807    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6a2197-2ddd-47d9-952c-aba449ba4d25-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:37:25.849917 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:25.849890    2572 scope.go:117] "RemoveContainer" containerID="6c5add39d78dbe626665b11023076cbc3b91ff37624f96949250f0570211063d"
Jul 17 20:37:26.519619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.519585    2572 generic.go:356] "Generic (PLEG): container finished" podID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerID="ba1ba3bab769a74945bb8898a25373a94a738fd802287d3ab4566f261307d321" exitCode=6
Jul 17 20:37:26.519809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.519657    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" event={"ID":"082ff1a9-ba00-4138-8881-4af88ad981a4","Type":"ContainerDied","Data":"ba1ba3bab769a74945bb8898a25373a94a738fd802287d3ab4566f261307d321"}
Jul 17 20:37:26.519809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.519713    2572 scope.go:117] "RemoveContainer" containerID="6c5add39d78dbe626665b11023076cbc3b91ff37624f96949250f0570211063d"
Jul 17 20:37:26.520038 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.520025    2572 scope.go:117] "RemoveContainer" containerID="ba1ba3bab769a74945bb8898a25373a94a738fd802287d3ab4566f261307d321"
Jul 17 20:37:26.520281 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:26.520255    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-xz9tr_odh-ai-gateway-infra(082ff1a9-ba00-4138-8881-4af88ad981a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4"
Jul 17 20:37:26.520986 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.520970    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"
Jul 17 20:37:26.521087 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.520987    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv" event={"ID":"5c6a2197-2ddd-47d9-952c-aba449ba4d25","Type":"ContainerDied","Data":"2a0a6e7f9b973f89e649093abe0f18c6b720db9cf106ebff6e61eb763f6b4bbe"}
Jul 17 20:37:26.530077 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.530060    2572 scope.go:117] "RemoveContainer" containerID="2e2810fd3b19b31a6aafd66e0f95b02990fb3ccdab5146bfb60d3cf5b6c14277"
Jul 17 20:37:26.548781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.548754    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"]
Jul 17 20:37:26.555499 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:26.555479    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lh2pv"]
Jul 17 20:37:27.547381 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:27.547349    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"]
Jul 17 20:37:27.681909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:27.681887    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"
Jul 17 20:37:27.849498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:27.849411    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082ff1a9-ba00-4138-8881-4af88ad981a4-maas-api-service-ca\") pod \"082ff1a9-ba00-4138-8881-4af88ad981a4\" (UID: \"082ff1a9-ba00-4138-8881-4af88ad981a4\") "
Jul 17 20:37:27.849793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:27.849773    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082ff1a9-ba00-4138-8881-4af88ad981a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "082ff1a9-ba00-4138-8881-4af88ad981a4" (UID: "082ff1a9-ba00-4138-8881-4af88ad981a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:37:27.853603 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:27.853580    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" path="/var/lib/kubelet/pods/5c6a2197-2ddd-47d9-952c-aba449ba4d25/volumes"
Jul 17 20:37:27.950491 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:27.950462    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082ff1a9-ba00-4138-8881-4af88ad981a4-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:37:28.534162 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:28.534131    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"
Jul 17 20:37:28.534363 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:28.534169    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr" event={"ID":"082ff1a9-ba00-4138-8881-4af88ad981a4","Type":"ContainerDied","Data":"97fe7b35b8c0dbfb0551b8a5a7aa10cb88161a8cf47290380c16e754691470e6"}
Jul 17 20:37:28.534363 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:28.534233    2572 scope.go:117] "RemoveContainer" containerID="ba1ba3bab769a74945bb8898a25373a94a738fd802287d3ab4566f261307d321"
Jul 17 20:37:28.549220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:28.549197    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"]
Jul 17 20:37:28.553967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:28.553947    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xz9tr"]
Jul 17 20:37:29.852908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:29.852876    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" path="/var/lib/kubelet/pods/082ff1a9-ba00-4138-8881-4af88ad981a4/volumes"
Jul 17 20:37:33.848093 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:33.848033    2572 scope.go:117] "RemoveContainer" containerID="7ca0ae830f76bcdb590e799d9c44d5f6dda27931d73f937b91ebae36d3d870cd"
Jul 17 20:37:34.562680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:34.562653    2572 generic.go:356] "Generic (PLEG): container finished" podID="546fa033-f3db-409f-83c5-171388682333" containerID="e52e8fc83b5f88af28a25e2380560cd11389c63677181fd1ea48e68f20b6d831" exitCode=6
Jul 17 20:37:34.562835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:34.562726    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" event={"ID":"546fa033-f3db-409f-83c5-171388682333","Type":"ContainerDied","Data":"e52e8fc83b5f88af28a25e2380560cd11389c63677181fd1ea48e68f20b6d831"}
Jul 17 20:37:34.562835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:34.562784    2572 scope.go:117] "RemoveContainer" containerID="7ca0ae830f76bcdb590e799d9c44d5f6dda27931d73f937b91ebae36d3d870cd"
Jul 17 20:37:34.563061 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:34.563031    2572 scope.go:117] "RemoveContainer" containerID="e52e8fc83b5f88af28a25e2380560cd11389c63677181fd1ea48e68f20b6d831"
Jul 17 20:37:34.563286 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:34.563266    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d_odh-ai-gateway-infra(546fa033-f3db-409f-83c5-171388682333)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" podUID="546fa033-f3db-409f-83c5-171388682333"
Jul 17 20:37:35.127748 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.127715    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"]
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128065    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128076    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128089    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128095    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128104    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128109    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128116    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:35.128132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128123    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:35.128411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128175    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:35.128411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128183    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:35.128411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128189    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:35.128411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128196    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:35.128411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.128202    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:35.132303 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.132287    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"
Jul 17 20:37:35.137013 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.136989    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"]
Jul 17 20:37:35.307764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.307735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79dee6fe-fa46-4cab-9929-39fb9842fae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf\" (UID: \"79dee6fe-fa46-4cab-9929-39fb9842fae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"
Jul 17 20:37:35.408963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.408862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79dee6fe-fa46-4cab-9929-39fb9842fae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf\" (UID: \"79dee6fe-fa46-4cab-9929-39fb9842fae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"
Jul 17 20:37:35.409457 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.409436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79dee6fe-fa46-4cab-9929-39fb9842fae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf\" (UID: \"79dee6fe-fa46-4cab-9929-39fb9842fae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"
Jul 17 20:37:35.443892 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.443860    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"
Jul 17 20:37:35.574091 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.574024    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"]
Jul 17 20:37:35.576234 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:37:35.576198    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79dee6fe_fa46_4cab_9929_39fb9842fae7.slice/crio-03531e9a5d4714e2b28578aa326c64a80bfd85fdaefb5fb2edb84005552c97ae WatchSource:0}: Error finding container 03531e9a5d4714e2b28578aa326c64a80bfd85fdaefb5fb2edb84005552c97ae: Status 404 returned error can't find the container with id 03531e9a5d4714e2b28578aa326c64a80bfd85fdaefb5fb2edb84005552c97ae
Jul 17 20:37:35.602238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.602212    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"]
Jul 17 20:37:35.729500 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.729478    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"
Jul 17 20:37:35.813351 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.813318    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546fa033-f3db-409f-83c5-171388682333-maas-api-service-ca\") pod \"546fa033-f3db-409f-83c5-171388682333\" (UID: \"546fa033-f3db-409f-83c5-171388682333\") "
Jul 17 20:37:35.813681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.813656    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546fa033-f3db-409f-83c5-171388682333-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "546fa033-f3db-409f-83c5-171388682333" (UID: "546fa033-f3db-409f-83c5-171388682333"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:37:35.914184 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:35.914152    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546fa033-f3db-409f-83c5-171388682333-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:37:36.574893 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.574860    2572 generic.go:356] "Generic (PLEG): container finished" podID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerID="6162b48ff271b1fab47380e088cb35db4bc064ce378191553cd24981ef596108" exitCode=6
Jul 17 20:37:36.575360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.574950    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" event={"ID":"79dee6fe-fa46-4cab-9929-39fb9842fae7","Type":"ContainerDied","Data":"6162b48ff271b1fab47380e088cb35db4bc064ce378191553cd24981ef596108"}
Jul 17 20:37:36.575360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.575013    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" event={"ID":"79dee6fe-fa46-4cab-9929-39fb9842fae7","Type":"ContainerStarted","Data":"03531e9a5d4714e2b28578aa326c64a80bfd85fdaefb5fb2edb84005552c97ae"}
Jul 17 20:37:36.575360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.575159    2572 scope.go:117] "RemoveContainer" containerID="6162b48ff271b1fab47380e088cb35db4bc064ce378191553cd24981ef596108"
Jul 17 20:37:36.576333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.576302    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d" event={"ID":"546fa033-f3db-409f-83c5-171388682333","Type":"ContainerDied","Data":"ed04d0acc01641ed43c72018cfe9ac518cde7b6d5b9f300b34e8ab54a4176e11"}
Jul 17 20:37:36.576333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.576323    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"
Jul 17 20:37:36.576501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.576356    2572 scope.go:117] "RemoveContainer" containerID="e52e8fc83b5f88af28a25e2380560cd11389c63677181fd1ea48e68f20b6d831"
Jul 17 20:37:36.611379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.611351    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"]
Jul 17 20:37:36.618358 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:36.618334    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rqf2d"]
Jul 17 20:37:37.582788 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:37.582756    2572 generic.go:356] "Generic (PLEG): container finished" podID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerID="3a40cb30be1cbd4bd42616d017b1466704307400714b8a29c5acb5ad77008a3b" exitCode=6
Jul 17 20:37:37.583274 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:37.582801    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" event={"ID":"79dee6fe-fa46-4cab-9929-39fb9842fae7","Type":"ContainerDied","Data":"3a40cb30be1cbd4bd42616d017b1466704307400714b8a29c5acb5ad77008a3b"}
Jul 17 20:37:37.583274 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:37.582855    2572 scope.go:117] "RemoveContainer" containerID="6162b48ff271b1fab47380e088cb35db4bc064ce378191553cd24981ef596108"
Jul 17 20:37:37.583274 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:37.583028    2572 scope.go:117] "RemoveContainer" containerID="3a40cb30be1cbd4bd42616d017b1466704307400714b8a29c5acb5ad77008a3b"
Jul 17 20:37:37.583274 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:37.583245    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf_odh-ai-gateway-infra(79dee6fe-fa46-4cab-9929-39fb9842fae7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7"
Jul 17 20:37:37.852787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:37.852712    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="546fa033-f3db-409f-83c5-171388682333" path="/var/lib/kubelet/pods/546fa033-f3db-409f-83c5-171388682333/volumes"
Jul 17 20:37:38.588513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:38.588485    2572 scope.go:117] "RemoveContainer" containerID="3a40cb30be1cbd4bd42616d017b1466704307400714b8a29c5acb5ad77008a3b"
Jul 17 20:37:38.588909 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:38.588670    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf_odh-ai-gateway-infra(79dee6fe-fa46-4cab-9929-39fb9842fae7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7"
Jul 17 20:37:51.848182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:51.848152    2572 scope.go:117] "RemoveContainer" containerID="3a40cb30be1cbd4bd42616d017b1466704307400714b8a29c5acb5ad77008a3b"
Jul 17 20:37:52.650548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:52.650509    2572 generic.go:356] "Generic (PLEG): container finished" podID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerID="85efb838fe3ffd089499d13236709265774bca5d99c4f7a6e3c81773133ad2c7" exitCode=6
Jul 17 20:37:52.650690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:52.650555    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" event={"ID":"79dee6fe-fa46-4cab-9929-39fb9842fae7","Type":"ContainerDied","Data":"85efb838fe3ffd089499d13236709265774bca5d99c4f7a6e3c81773133ad2c7"}
Jul 17 20:37:52.650690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:52.650601    2572 scope.go:117] "RemoveContainer" containerID="3a40cb30be1cbd4bd42616d017b1466704307400714b8a29c5acb5ad77008a3b"
Jul 17 20:37:52.650860 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:52.650844    2572 scope.go:117] "RemoveContainer" containerID="85efb838fe3ffd089499d13236709265774bca5d99c4f7a6e3c81773133ad2c7"
Jul 17 20:37:52.651063 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:37:52.651024    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf_odh-ai-gateway-infra(79dee6fe-fa46-4cab-9929-39fb9842fae7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7"
Jul 17 20:37:53.677815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:53.677782    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"]
Jul 17 20:37:53.810119 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:53.810097    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"
Jul 17 20:37:53.853226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:53.853195    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79dee6fe-fa46-4cab-9929-39fb9842fae7-maas-api-service-ca\") pod \"79dee6fe-fa46-4cab-9929-39fb9842fae7\" (UID: \"79dee6fe-fa46-4cab-9929-39fb9842fae7\") "
Jul 17 20:37:53.853539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:53.853517    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79dee6fe-fa46-4cab-9929-39fb9842fae7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79dee6fe-fa46-4cab-9929-39fb9842fae7" (UID: "79dee6fe-fa46-4cab-9929-39fb9842fae7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:37:53.954643 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:53.954557    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79dee6fe-fa46-4cab-9929-39fb9842fae7-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:37:54.662742 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:54.662707    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"
Jul 17 20:37:54.662742 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:54.662723    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf" event={"ID":"79dee6fe-fa46-4cab-9929-39fb9842fae7","Type":"ContainerDied","Data":"03531e9a5d4714e2b28578aa326c64a80bfd85fdaefb5fb2edb84005552c97ae"}
Jul 17 20:37:54.662947 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:54.662787    2572 scope.go:117] "RemoveContainer" containerID="85efb838fe3ffd089499d13236709265774bca5d99c4f7a6e3c81773133ad2c7"
Jul 17 20:37:54.680098 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:54.680070    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"]
Jul 17 20:37:54.683897 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:54.683877    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-9lbcf"]
Jul 17 20:37:55.853387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:55.853350    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" path="/var/lib/kubelet/pods/79dee6fe-fa46-4cab-9929-39fb9842fae7/volumes"
Jul 17 20:37:59.240549 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240469    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"]
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240766    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240775    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240785    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240790    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240798    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240803    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240818    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240824    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="082ff1a9-ba00-4138-8881-4af88ad981a4" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240830    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240835    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240841    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240845    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240851    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240857    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240863    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240868    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:59.240907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240910    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.241471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240917    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6a2197-2ddd-47d9-952c-aba449ba4d25" containerName="revoke-keys"
Jul 17 20:37:59.241471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240923    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.241471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240931    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:37:59.241471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240936    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.241471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.240941    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="79dee6fe-fa46-4cab-9929-39fb9842fae7" containerName="revoke-keys"
Jul 17 20:37:59.245262 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.245234    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"
Jul 17 20:37:59.250585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.250564    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:37:59.250854 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.250838    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:37:59.266532 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.266499    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"]
Jul 17 20:37:59.295864 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.295837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef007625-5403-403f-89b3-8eeac38b27c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx\" (UID: \"ef007625-5403-403f-89b3-8eeac38b27c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"
Jul 17 20:37:59.396660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.396628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef007625-5403-403f-89b3-8eeac38b27c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx\" (UID: \"ef007625-5403-403f-89b3-8eeac38b27c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"
Jul 17 20:37:59.397213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.397196    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef007625-5403-403f-89b3-8eeac38b27c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx\" (UID: \"ef007625-5403-403f-89b3-8eeac38b27c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"
Jul 17 20:37:59.554985 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.554901    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"
Jul 17 20:37:59.704552 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:37:59.704519    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"]
Jul 17 20:37:59.707464 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:37:59.707436    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef007625_5403_403f_89b3_8eeac38b27c8.slice/crio-a307dd46b64188447a8914661c5d9dd24dc397392a3a3b5a08b48ba8d79bd2c6 WatchSource:0}: Error finding container a307dd46b64188447a8914661c5d9dd24dc397392a3a3b5a08b48ba8d79bd2c6: Status 404 returned error can't find the container with id a307dd46b64188447a8914661c5d9dd24dc397392a3a3b5a08b48ba8d79bd2c6
Jul 17 20:38:00.690343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:00.690259    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef007625-5403-403f-89b3-8eeac38b27c8" containerID="27a7c2497d5c2360570f070c3d9c10e9897f9d75046db6e351fb1dff2d955c46" exitCode=6
Jul 17 20:38:00.690343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:00.690287    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" event={"ID":"ef007625-5403-403f-89b3-8eeac38b27c8","Type":"ContainerDied","Data":"27a7c2497d5c2360570f070c3d9c10e9897f9d75046db6e351fb1dff2d955c46"}
Jul 17 20:38:00.690724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:00.690345    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" event={"ID":"ef007625-5403-403f-89b3-8eeac38b27c8","Type":"ContainerStarted","Data":"a307dd46b64188447a8914661c5d9dd24dc397392a3a3b5a08b48ba8d79bd2c6"}
Jul 17 20:38:00.690724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:00.690602    2572 scope.go:117] "RemoveContainer" containerID="27a7c2497d5c2360570f070c3d9c10e9897f9d75046db6e351fb1dff2d955c46"
Jul 17 20:38:01.147702 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.147670    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"]
Jul 17 20:38:01.148022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.148011    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="546fa033-f3db-409f-83c5-171388682333" containerName="revoke-keys"
Jul 17 20:38:01.150784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.150763    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"
Jul 17 20:38:01.164597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.164310    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"]
Jul 17 20:38:01.211057 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.211024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55324a7f-1ac8-4949-b8b5-a2c3274bf9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-82zx4\" (UID: \"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"
Jul 17 20:38:01.311572 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.311545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55324a7f-1ac8-4949-b8b5-a2c3274bf9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-82zx4\" (UID: \"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"
Jul 17 20:38:01.312129 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.312112    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55324a7f-1ac8-4949-b8b5-a2c3274bf9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-82zx4\" (UID: \"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"
Jul 17 20:38:01.467841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.467813    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"
Jul 17 20:38:01.590390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.590365    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"]
Jul 17 20:38:01.592900 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:38:01.592876    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55324a7f_1ac8_4949_b8b5_a2c3274bf9bc.slice/crio-c315247e728175aefa948141f929bbe3579c60f0db0f8eac0fd6faae22498894 WatchSource:0}: Error finding container c315247e728175aefa948141f929bbe3579c60f0db0f8eac0fd6faae22498894: Status 404 returned error can't find the container with id c315247e728175aefa948141f929bbe3579c60f0db0f8eac0fd6faae22498894
Jul 17 20:38:01.696377 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.696345    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef007625-5403-403f-89b3-8eeac38b27c8" containerID="4da681d10ec4cb416fdc6ca6d9bd8821d5b818b0a81e5c0ce2868b90d927af87" exitCode=6
Jul 17 20:38:01.696804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.696425    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" event={"ID":"ef007625-5403-403f-89b3-8eeac38b27c8","Type":"ContainerDied","Data":"4da681d10ec4cb416fdc6ca6d9bd8821d5b818b0a81e5c0ce2868b90d927af87"}
Jul 17 20:38:01.696804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.696491    2572 scope.go:117] "RemoveContainer" containerID="27a7c2497d5c2360570f070c3d9c10e9897f9d75046db6e351fb1dff2d955c46"
Jul 17 20:38:01.696804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.696715    2572 scope.go:117] "RemoveContainer" containerID="4da681d10ec4cb416fdc6ca6d9bd8821d5b818b0a81e5c0ce2868b90d927af87"
Jul 17 20:38:01.696993 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:01.696973    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx_odh-ai-gateway-infra(ef007625-5403-403f-89b3-8eeac38b27c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" podUID="ef007625-5403-403f-89b3-8eeac38b27c8"
Jul 17 20:38:01.697798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:01.697675    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" event={"ID":"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc","Type":"ContainerStarted","Data":"c315247e728175aefa948141f929bbe3579c60f0db0f8eac0fd6faae22498894"}
Jul 17 20:38:02.703510 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:02.703481    2572 scope.go:117] "RemoveContainer" containerID="4da681d10ec4cb416fdc6ca6d9bd8821d5b818b0a81e5c0ce2868b90d927af87"
Jul 17 20:38:02.703950 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:02.703694    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx_odh-ai-gateway-infra(ef007625-5403-403f-89b3-8eeac38b27c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" podUID="ef007625-5403-403f-89b3-8eeac38b27c8"
Jul 17 20:38:02.704599 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:02.704576    2572 generic.go:356] "Generic (PLEG): container finished" podID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerID="bdfe6330c09e66d0f3ca9fb38129933e910ec623c949d0912beb0e3667dfc0d9" exitCode=6
Jul 17 20:38:02.704684 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:02.704625    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" event={"ID":"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc","Type":"ContainerDied","Data":"bdfe6330c09e66d0f3ca9fb38129933e910ec623c949d0912beb0e3667dfc0d9"}
Jul 17 20:38:02.704791 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:02.704779    2572 scope.go:117] "RemoveContainer" containerID="bdfe6330c09e66d0f3ca9fb38129933e910ec623c949d0912beb0e3667dfc0d9"
Jul 17 20:38:03.711014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:03.710977    2572 generic.go:356] "Generic (PLEG): container finished" podID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerID="1a88617b9103887a8002066140f36a74bf25baa457cf81e5525de05584e87a11" exitCode=6
Jul 17 20:38:03.711439 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:03.711039    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" event={"ID":"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc","Type":"ContainerDied","Data":"1a88617b9103887a8002066140f36a74bf25baa457cf81e5525de05584e87a11"}
Jul 17 20:38:03.711439 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:03.711111    2572 scope.go:117] "RemoveContainer" containerID="bdfe6330c09e66d0f3ca9fb38129933e910ec623c949d0912beb0e3667dfc0d9"
Jul 17 20:38:03.711439 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:03.711263    2572 scope.go:117] "RemoveContainer" containerID="1a88617b9103887a8002066140f36a74bf25baa457cf81e5525de05584e87a11"
Jul 17 20:38:03.711555 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:03.711453    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-82zx4_odh-ai-gateway-infra(55324a7f-1ac8-4949-b8b5-a2c3274bf9bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc"
Jul 17 20:38:04.717711 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:04.717687    2572 scope.go:117] "RemoveContainer" containerID="1a88617b9103887a8002066140f36a74bf25baa457cf81e5525de05584e87a11"
Jul 17 20:38:04.718122 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:04.717865    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-82zx4_odh-ai-gateway-infra(55324a7f-1ac8-4949-b8b5-a2c3274bf9bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc"
Jul 17 20:38:07.349475 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.349440    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"]
Jul 17 20:38:07.354202 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.354185    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"
Jul 17 20:38:07.359757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.359731    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"]
Jul 17 20:38:07.454762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.454728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743f90f-ba79-4fe1-87cb-cbea8ed901e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk\" (UID: \"6743f90f-ba79-4fe1-87cb-cbea8ed901e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"
Jul 17 20:38:07.556000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.555957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743f90f-ba79-4fe1-87cb-cbea8ed901e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk\" (UID: \"6743f90f-ba79-4fe1-87cb-cbea8ed901e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"
Jul 17 20:38:07.556643 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.556615    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743f90f-ba79-4fe1-87cb-cbea8ed901e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk\" (UID: \"6743f90f-ba79-4fe1-87cb-cbea8ed901e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"
Jul 17 20:38:07.666493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.666403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"
Jul 17 20:38:07.794242 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:07.794214    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"]
Jul 17 20:38:07.797092 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:38:07.797067    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6743f90f_ba79_4fe1_87cb_cbea8ed901e3.slice/crio-486ae5728f8ce97885efc472d29d62baa3c2de0d8b67ab85924cd3d2cd640980 WatchSource:0}: Error finding container 486ae5728f8ce97885efc472d29d62baa3c2de0d8b67ab85924cd3d2cd640980: Status 404 returned error can't find the container with id 486ae5728f8ce97885efc472d29d62baa3c2de0d8b67ab85924cd3d2cd640980
Jul 17 20:38:08.735703 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:08.735668    2572 generic.go:356] "Generic (PLEG): container finished" podID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerID="c761d5140ca183d8279340fbf2c82de33145983318fcaef6ad1d6e0bb86a78f1" exitCode=6
Jul 17 20:38:08.736099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:08.735733    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" event={"ID":"6743f90f-ba79-4fe1-87cb-cbea8ed901e3","Type":"ContainerDied","Data":"c761d5140ca183d8279340fbf2c82de33145983318fcaef6ad1d6e0bb86a78f1"}
Jul 17 20:38:08.736099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:08.735782    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" event={"ID":"6743f90f-ba79-4fe1-87cb-cbea8ed901e3","Type":"ContainerStarted","Data":"486ae5728f8ce97885efc472d29d62baa3c2de0d8b67ab85924cd3d2cd640980"}
Jul 17 20:38:08.736099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:08.736002    2572 scope.go:117] "RemoveContainer" containerID="c761d5140ca183d8279340fbf2c82de33145983318fcaef6ad1d6e0bb86a78f1"
Jul 17 20:38:09.741823 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:09.741789    2572 generic.go:356] "Generic (PLEG): container finished" podID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerID="fb5ce57bfb07b9d40604ec51ca639914ecf1bc12f843794d3f1e08b8b54d37c0" exitCode=6
Jul 17 20:38:09.742312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:09.741874    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" event={"ID":"6743f90f-ba79-4fe1-87cb-cbea8ed901e3","Type":"ContainerDied","Data":"fb5ce57bfb07b9d40604ec51ca639914ecf1bc12f843794d3f1e08b8b54d37c0"}
Jul 17 20:38:09.742312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:09.741949    2572 scope.go:117] "RemoveContainer" containerID="c761d5140ca183d8279340fbf2c82de33145983318fcaef6ad1d6e0bb86a78f1"
Jul 17 20:38:09.742312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:09.742191    2572 scope.go:117] "RemoveContainer" containerID="fb5ce57bfb07b9d40604ec51ca639914ecf1bc12f843794d3f1e08b8b54d37c0"
Jul 17 20:38:09.742490 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:09.742387    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk_odh-ai-gateway-infra(6743f90f-ba79-4fe1-87cb-cbea8ed901e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3"
Jul 17 20:38:10.748026 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:10.747998    2572 scope.go:117] "RemoveContainer" containerID="fb5ce57bfb07b9d40604ec51ca639914ecf1bc12f843794d3f1e08b8b54d37c0"
Jul 17 20:38:10.748398 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:10.748197    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk_odh-ai-gateway-infra(6743f90f-ba79-4fe1-87cb-cbea8ed901e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3"
Jul 17 20:38:15.850282 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:15.850246    2572 scope.go:117] "RemoveContainer" containerID="1a88617b9103887a8002066140f36a74bf25baa457cf81e5525de05584e87a11"
Jul 17 20:38:16.774112 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:16.774082    2572 generic.go:356] "Generic (PLEG): container finished" podID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerID="78f9f6c862189aa5f1d36f4a16001e259cde1ed9f633099e3f6e16bb781f9fb5" exitCode=6
Jul 17 20:38:16.774281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:16.774151    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" event={"ID":"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc","Type":"ContainerDied","Data":"78f9f6c862189aa5f1d36f4a16001e259cde1ed9f633099e3f6e16bb781f9fb5"}
Jul 17 20:38:16.774281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:16.774207    2572 scope.go:117] "RemoveContainer" containerID="1a88617b9103887a8002066140f36a74bf25baa457cf81e5525de05584e87a11"
Jul 17 20:38:16.774487 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:16.774472    2572 scope.go:117] "RemoveContainer" containerID="78f9f6c862189aa5f1d36f4a16001e259cde1ed9f633099e3f6e16bb781f9fb5"
Jul 17 20:38:16.774696 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:16.774679    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-82zx4_odh-ai-gateway-infra(55324a7f-1ac8-4949-b8b5-a2c3274bf9bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc"
Jul 17 20:38:17.798711 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:17.798673    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"]
Jul 17 20:38:17.848471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:17.848446    2572 scope.go:117] "RemoveContainer" containerID="4da681d10ec4cb416fdc6ca6d9bd8821d5b818b0a81e5c0ce2868b90d927af87"
Jul 17 20:38:17.929102 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:17.929081    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"
Jul 17 20:38:18.036748 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.036717    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55324a7f-1ac8-4949-b8b5-a2c3274bf9bc-maas-api-service-ca\") pod \"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc\" (UID: \"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc\") "
Jul 17 20:38:18.037005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.036985    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55324a7f-1ac8-4949-b8b5-a2c3274bf9bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" (UID: "55324a7f-1ac8-4949-b8b5-a2c3274bf9bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:38:18.137708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.137645    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55324a7f-1ac8-4949-b8b5-a2c3274bf9bc-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:38:18.786068 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.785967    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef007625-5403-403f-89b3-8eeac38b27c8" containerID="75ad016892d11419648cd2a156bb9544d9d1e30cf4b4ae7f51fb75b5b7283bd0" exitCode=6
Jul 17 20:38:18.786258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.786071    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" event={"ID":"ef007625-5403-403f-89b3-8eeac38b27c8","Type":"ContainerDied","Data":"75ad016892d11419648cd2a156bb9544d9d1e30cf4b4ae7f51fb75b5b7283bd0"}
Jul 17 20:38:18.786258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.786133    2572 scope.go:117] "RemoveContainer" containerID="4da681d10ec4cb416fdc6ca6d9bd8821d5b818b0a81e5c0ce2868b90d927af87"
Jul 17 20:38:18.786440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.786421    2572 scope.go:117] "RemoveContainer" containerID="75ad016892d11419648cd2a156bb9544d9d1e30cf4b4ae7f51fb75b5b7283bd0"
Jul 17 20:38:18.786682 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:18.786659    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx_odh-ai-gateway-infra(ef007625-5403-403f-89b3-8eeac38b27c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" podUID="ef007625-5403-403f-89b3-8eeac38b27c8"
Jul 17 20:38:18.787394 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.787373    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4" event={"ID":"55324a7f-1ac8-4949-b8b5-a2c3274bf9bc","Type":"ContainerDied","Data":"c315247e728175aefa948141f929bbe3579c60f0db0f8eac0fd6faae22498894"}
Jul 17 20:38:18.787394 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.787385    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"
Jul 17 20:38:18.796888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.796871    2572 scope.go:117] "RemoveContainer" containerID="78f9f6c862189aa5f1d36f4a16001e259cde1ed9f633099e3f6e16bb781f9fb5"
Jul 17 20:38:18.814121 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.814101    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"]
Jul 17 20:38:18.818652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:18.818631    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-82zx4"]
Jul 17 20:38:19.814315 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:19.814284    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"]
Jul 17 20:38:19.853176 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:19.853145    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" path="/var/lib/kubelet/pods/55324a7f-1ac8-4949-b8b5-a2c3274bf9bc/volumes"
Jul 17 20:38:19.948752 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:19.948726    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"
Jul 17 20:38:20.052837 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.052807    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef007625-5403-403f-89b3-8eeac38b27c8-maas-api-service-ca\") pod \"ef007625-5403-403f-89b3-8eeac38b27c8\" (UID: \"ef007625-5403-403f-89b3-8eeac38b27c8\") "
Jul 17 20:38:20.053200 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.053182    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef007625-5403-403f-89b3-8eeac38b27c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef007625-5403-403f-89b3-8eeac38b27c8" (UID: "ef007625-5403-403f-89b3-8eeac38b27c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:38:20.153335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.153252    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef007625-5403-403f-89b3-8eeac38b27c8-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:38:20.799625 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.799587    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx" event={"ID":"ef007625-5403-403f-89b3-8eeac38b27c8","Type":"ContainerDied","Data":"a307dd46b64188447a8914661c5d9dd24dc397392a3a3b5a08b48ba8d79bd2c6"}
Jul 17 20:38:20.799625 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.799615    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"
Jul 17 20:38:20.799885 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.799676    2572 scope.go:117] "RemoveContainer" containerID="75ad016892d11419648cd2a156bb9544d9d1e30cf4b4ae7f51fb75b5b7283bd0"
Jul 17 20:38:20.819358 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.819336    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"]
Jul 17 20:38:20.825008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:20.824987    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-5h7nx"]
Jul 17 20:38:21.853096 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:21.853041    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" path="/var/lib/kubelet/pods/ef007625-5403-403f-89b3-8eeac38b27c8/volumes"
Jul 17 20:38:24.848096 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:24.848070    2572 scope.go:117] "RemoveContainer" containerID="fb5ce57bfb07b9d40604ec51ca639914ecf1bc12f843794d3f1e08b8b54d37c0"
Jul 17 20:38:25.176730 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.176697    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"]
Jul 17 20:38:25.177008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.176995    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177009    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177022    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177027    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177036    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177063    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177072    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177078    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177087    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177092    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177138    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177147    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177158    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef007625-5403-403f-89b3-8eeac38b27c8" containerName="revoke-keys"
Jul 17 20:38:25.177407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177168    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:25.177407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177174    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:25.177407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.177180    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:25.180257 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.180240    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"
Jul 17 20:38:25.186425 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.186369    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"]
Jul 17 20:38:25.296650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.296617    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7148eccf-23f3-40a8-8916-f2be80c2be40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk\" (UID: \"7148eccf-23f3-40a8-8916-f2be80c2be40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"
Jul 17 20:38:25.397675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.397584    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7148eccf-23f3-40a8-8916-f2be80c2be40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk\" (UID: \"7148eccf-23f3-40a8-8916-f2be80c2be40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"
Jul 17 20:38:25.398281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.398263    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7148eccf-23f3-40a8-8916-f2be80c2be40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk\" (UID: \"7148eccf-23f3-40a8-8916-f2be80c2be40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"
Jul 17 20:38:25.491554 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.491523    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"
Jul 17 20:38:25.824938 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.824915    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"]
Jul 17 20:38:25.825660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.825637    2572 generic.go:356] "Generic (PLEG): container finished" podID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerID="1cb82c49217131ffb61564023328d904220af65f8a4e5835bebecca2f1e6cbd4" exitCode=6
Jul 17 20:38:25.825750 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.825711    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" event={"ID":"6743f90f-ba79-4fe1-87cb-cbea8ed901e3","Type":"ContainerDied","Data":"1cb82c49217131ffb61564023328d904220af65f8a4e5835bebecca2f1e6cbd4"}
Jul 17 20:38:25.825792 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.825779    2572 scope.go:117] "RemoveContainer" containerID="fb5ce57bfb07b9d40604ec51ca639914ecf1bc12f843794d3f1e08b8b54d37c0"
Jul 17 20:38:25.826189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:25.826168    2572 scope.go:117] "RemoveContainer" containerID="1cb82c49217131ffb61564023328d904220af65f8a4e5835bebecca2f1e6cbd4"
Jul 17 20:38:25.826437 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:25.826415    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk_odh-ai-gateway-infra(6743f90f-ba79-4fe1-87cb-cbea8ed901e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3"
Jul 17 20:38:25.827516 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:38:25.827493    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7148eccf_23f3_40a8_8916_f2be80c2be40.slice/crio-008668103df6ce28c305ee451803066a97c6aaf90f47b28ac51f019f14ad63a0 WatchSource:0}: Error finding container 008668103df6ce28c305ee451803066a97c6aaf90f47b28ac51f019f14ad63a0: Status 404 returned error can't find the container with id 008668103df6ce28c305ee451803066a97c6aaf90f47b28ac51f019f14ad63a0
Jul 17 20:38:26.831630 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:26.831597    2572 generic.go:356] "Generic (PLEG): container finished" podID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerID="2f9f8d5dca799babfd561d808035553857d7d92ca5d1b31003941dc8fd5f1090" exitCode=6
Jul 17 20:38:26.832079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:26.831682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" event={"ID":"7148eccf-23f3-40a8-8916-f2be80c2be40","Type":"ContainerDied","Data":"2f9f8d5dca799babfd561d808035553857d7d92ca5d1b31003941dc8fd5f1090"}
Jul 17 20:38:26.832079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:26.831776    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" event={"ID":"7148eccf-23f3-40a8-8916-f2be80c2be40","Type":"ContainerStarted","Data":"008668103df6ce28c305ee451803066a97c6aaf90f47b28ac51f019f14ad63a0"}
Jul 17 20:38:26.832079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:26.831825    2572 scope.go:117] "RemoveContainer" containerID="2f9f8d5dca799babfd561d808035553857d7d92ca5d1b31003941dc8fd5f1090"
Jul 17 20:38:26.851619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:26.851598    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"]
Jul 17 20:38:26.979029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:26.979005    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"
Jul 17 20:38:27.113702 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.113625    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743f90f-ba79-4fe1-87cb-cbea8ed901e3-maas-api-service-ca\") pod \"6743f90f-ba79-4fe1-87cb-cbea8ed901e3\" (UID: \"6743f90f-ba79-4fe1-87cb-cbea8ed901e3\") "
Jul 17 20:38:27.113989 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.113970    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6743f90f-ba79-4fe1-87cb-cbea8ed901e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6743f90f-ba79-4fe1-87cb-cbea8ed901e3" (UID: "6743f90f-ba79-4fe1-87cb-cbea8ed901e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:38:27.215157 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.215127    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743f90f-ba79-4fe1-87cb-cbea8ed901e3-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:38:27.838419 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.838390    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"
Jul 17 20:38:27.838896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.838426    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk" event={"ID":"6743f90f-ba79-4fe1-87cb-cbea8ed901e3","Type":"ContainerDied","Data":"486ae5728f8ce97885efc472d29d62baa3c2de0d8b67ab85924cd3d2cd640980"}
Jul 17 20:38:27.838896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.838496    2572 scope.go:117] "RemoveContainer" containerID="1cb82c49217131ffb61564023328d904220af65f8a4e5835bebecca2f1e6cbd4"
Jul 17 20:38:27.840255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.840232    2572 generic.go:356] "Generic (PLEG): container finished" podID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerID="9e37ce4e17c4fff9c8b1ca5ecdb05c11ac854f8a1b7e5c06ad941a3b49cf1ae6" exitCode=6
Jul 17 20:38:27.840364 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.840273    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" event={"ID":"7148eccf-23f3-40a8-8916-f2be80c2be40","Type":"ContainerDied","Data":"9e37ce4e17c4fff9c8b1ca5ecdb05c11ac854f8a1b7e5c06ad941a3b49cf1ae6"}
Jul 17 20:38:27.840536 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.840521    2572 scope.go:117] "RemoveContainer" containerID="9e37ce4e17c4fff9c8b1ca5ecdb05c11ac854f8a1b7e5c06ad941a3b49cf1ae6"
Jul 17 20:38:27.840763 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:27.840741    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk_odh-ai-gateway-infra(7148eccf-23f3-40a8-8916-f2be80c2be40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40"
Jul 17 20:38:27.847301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.847284    2572 scope.go:117] "RemoveContainer" containerID="2f9f8d5dca799babfd561d808035553857d7d92ca5d1b31003941dc8fd5f1090"
Jul 17 20:38:27.871445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.871422    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"]
Jul 17 20:38:27.876857 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:27.876835    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5ttpk"]
Jul 17 20:38:28.846792 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:28.846764    2572 scope.go:117] "RemoveContainer" containerID="9e37ce4e17c4fff9c8b1ca5ecdb05c11ac854f8a1b7e5c06ad941a3b49cf1ae6"
Jul 17 20:38:28.847188 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:28.846936    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk_odh-ai-gateway-infra(7148eccf-23f3-40a8-8916-f2be80c2be40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40"
Jul 17 20:38:29.852608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:29.852576    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" path="/var/lib/kubelet/pods/6743f90f-ba79-4fe1-87cb-cbea8ed901e3/volumes"
Jul 17 20:38:39.360564 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:39.360525    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:38:41.848268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:41.848236    2572 scope.go:117] "RemoveContainer" containerID="9e37ce4e17c4fff9c8b1ca5ecdb05c11ac854f8a1b7e5c06ad941a3b49cf1ae6"
Jul 17 20:38:42.904715 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:42.904685    2572 generic.go:356] "Generic (PLEG): container finished" podID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerID="a776cc5f301f35897ba6b1206864abadfe0d05e5be5ddc6b335c49c5e54a8813" exitCode=6
Jul 17 20:38:42.905153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:42.904735    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" event={"ID":"7148eccf-23f3-40a8-8916-f2be80c2be40","Type":"ContainerDied","Data":"a776cc5f301f35897ba6b1206864abadfe0d05e5be5ddc6b335c49c5e54a8813"}
Jul 17 20:38:42.905153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:42.904782    2572 scope.go:117] "RemoveContainer" containerID="9e37ce4e17c4fff9c8b1ca5ecdb05c11ac854f8a1b7e5c06ad941a3b49cf1ae6"
Jul 17 20:38:42.905153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:42.905038    2572 scope.go:117] "RemoveContainer" containerID="a776cc5f301f35897ba6b1206864abadfe0d05e5be5ddc6b335c49c5e54a8813"
Jul 17 20:38:42.905277 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:42.905258    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk_odh-ai-gateway-infra(7148eccf-23f3-40a8-8916-f2be80c2be40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40"
Jul 17 20:38:43.929839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:43.929805    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"]
Jul 17 20:38:44.061951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.061924    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"
Jul 17 20:38:44.146975 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.146943    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7148eccf-23f3-40a8-8916-f2be80c2be40-maas-api-service-ca\") pod \"7148eccf-23f3-40a8-8916-f2be80c2be40\" (UID: \"7148eccf-23f3-40a8-8916-f2be80c2be40\") "
Jul 17 20:38:44.147297 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.147272    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7148eccf-23f3-40a8-8916-f2be80c2be40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7148eccf-23f3-40a8-8916-f2be80c2be40" (UID: "7148eccf-23f3-40a8-8916-f2be80c2be40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:38:44.248252 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.248218    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7148eccf-23f3-40a8-8916-f2be80c2be40-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:38:44.921064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.921025    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"
Jul 17 20:38:44.921216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.921025    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk" event={"ID":"7148eccf-23f3-40a8-8916-f2be80c2be40","Type":"ContainerDied","Data":"008668103df6ce28c305ee451803066a97c6aaf90f47b28ac51f019f14ad63a0"}
Jul 17 20:38:44.921216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.921202    2572 scope.go:117] "RemoveContainer" containerID="a776cc5f301f35897ba6b1206864abadfe0d05e5be5ddc6b335c49c5e54a8813"
Jul 17 20:38:44.942022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.941993    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"]
Jul 17 20:38:44.948744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:44.948720    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-qm5fk"]
Jul 17 20:38:45.852731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:45.852701    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" path="/var/lib/kubelet/pods/7148eccf-23f3-40a8-8916-f2be80c2be40/volumes"
Jul 17 20:38:46.798096 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:46.798059    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:38:51.190103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190071    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"]
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190366    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190376    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190386    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190392    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190401    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190408    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190416    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190421    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190428    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190433    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190438    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:51.190479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190444    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="55324a7f-1ac8-4949-b8b5-a2c3274bf9bc" containerName="revoke-keys"
Jul 17 20:38:51.190856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190512    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:51.190856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190534    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:51.190856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190540    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190547    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.190856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.190560    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7148eccf-23f3-40a8-8916-f2be80c2be40" containerName="revoke-keys"
Jul 17 20:38:51.194709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.194685    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"
Jul 17 20:38:51.197513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.197493    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:38:51.197623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.197493    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:38:51.203308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.203285    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"]
Jul 17 20:38:51.312556 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.312523    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-pjsmq\" (UID: \"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"
Jul 17 20:38:51.413021 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.412989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-pjsmq\" (UID: \"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"
Jul 17 20:38:51.413664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.413638    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-pjsmq\" (UID: \"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"
Jul 17 20:38:51.506312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.506272    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"
Jul 17 20:38:51.638333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.638308    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"]
Jul 17 20:38:51.640765 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:38:51.640740    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a8de07_8b18_4c6f_9ca8_6a9f858cd82b.slice/crio-13b251cee1e36d50b0c30754b389a853d80f7caeed9b4f1e259ad72b9a16a7d2 WatchSource:0}: Error finding container 13b251cee1e36d50b0c30754b389a853d80f7caeed9b4f1e259ad72b9a16a7d2: Status 404 returned error can't find the container with id 13b251cee1e36d50b0c30754b389a853d80f7caeed9b4f1e259ad72b9a16a7d2
Jul 17 20:38:51.953510 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.953479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" event={"ID":"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b","Type":"ContainerStarted","Data":"69682f4328ce0613a19ea9d49f477894bf960666b40bf423f4d7444a6b741d64"}
Jul 17 20:38:51.953670 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.953528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" event={"ID":"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b","Type":"ContainerStarted","Data":"13b251cee1e36d50b0c30754b389a853d80f7caeed9b4f1e259ad72b9a16a7d2"}
Jul 17 20:38:51.969990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:51.969948    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" podStartSLOduration=0.969932224 podStartE2EDuration="969.932224ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:38:51.968292156 +0000 UTC m=+2566.662268306" watchObservedRunningTime="2026-07-17 20:38:51.969932224 +0000 UTC m=+2566.663908348"
Jul 17 20:38:52.959745 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:52.959710    2572 generic.go:356] "Generic (PLEG): container finished" podID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerID="69682f4328ce0613a19ea9d49f477894bf960666b40bf423f4d7444a6b741d64" exitCode=6
Jul 17 20:38:52.960173 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:52.959756    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" event={"ID":"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b","Type":"ContainerDied","Data":"69682f4328ce0613a19ea9d49f477894bf960666b40bf423f4d7444a6b741d64"}
Jul 17 20:38:52.960173 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:52.960011    2572 scope.go:117] "RemoveContainer" containerID="69682f4328ce0613a19ea9d49f477894bf960666b40bf423f4d7444a6b741d64"
Jul 17 20:38:53.965943 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:53.965904    2572 generic.go:356] "Generic (PLEG): container finished" podID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerID="e7a2389867e43a4351e46a7680f1ff346286e51dcc8c150067b75d013d4dcf93" exitCode=6
Jul 17 20:38:53.966434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:53.965960    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" event={"ID":"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b","Type":"ContainerDied","Data":"e7a2389867e43a4351e46a7680f1ff346286e51dcc8c150067b75d013d4dcf93"}
Jul 17 20:38:53.966434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:53.966005    2572 scope.go:117] "RemoveContainer" containerID="69682f4328ce0613a19ea9d49f477894bf960666b40bf423f4d7444a6b741d64"
Jul 17 20:38:53.966434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:53.966242    2572 scope.go:117] "RemoveContainer" containerID="e7a2389867e43a4351e46a7680f1ff346286e51dcc8c150067b75d013d4dcf93"
Jul 17 20:38:53.966606 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:53.966465    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-pjsmq_odh-ai-gateway-infra(d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b"
Jul 17 20:38:54.972265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:54.972231    2572 scope.go:117] "RemoveContainer" containerID="e7a2389867e43a4351e46a7680f1ff346286e51dcc8c150067b75d013d4dcf93"
Jul 17 20:38:54.972737 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:38:54.972443    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-pjsmq_odh-ai-gateway-infra(d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b"
Jul 17 20:38:55.796109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:55.796075    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:38:59.281943 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.281906    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"]
Jul 17 20:38:59.282565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.282379    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:59.282565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.282399    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:59.282565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.282480    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6743f90f-ba79-4fe1-87cb-cbea8ed901e3" containerName="revoke-keys"
Jul 17 20:38:59.286864 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.286843    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"
Jul 17 20:38:59.292411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.292386    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"]
Jul 17 20:38:59.375835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.375797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7782764-acee-4624-9eac-97719f137ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp\" (UID: \"c7782764-acee-4624-9eac-97719f137ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"
Jul 17 20:38:59.477178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.477139    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7782764-acee-4624-9eac-97719f137ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp\" (UID: \"c7782764-acee-4624-9eac-97719f137ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"
Jul 17 20:38:59.477755 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.477733    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7782764-acee-4624-9eac-97719f137ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp\" (UID: \"c7782764-acee-4624-9eac-97719f137ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"
Jul 17 20:38:59.598038 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.597941    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"
Jul 17 20:38:59.730201 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.730172    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"]
Jul 17 20:38:59.732165 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:38:59.732135    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7782764_acee_4624_9eac_97719f137ffd.slice/crio-61ec32afdb8529681bc27eb0bba495812818c9f916e3fa376a0ce3c04ce44dc3 WatchSource:0}: Error finding container 61ec32afdb8529681bc27eb0bba495812818c9f916e3fa376a0ce3c04ce44dc3: Status 404 returned error can't find the container with id 61ec32afdb8529681bc27eb0bba495812818c9f916e3fa376a0ce3c04ce44dc3
Jul 17 20:38:59.994787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:38:59.994751    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" event={"ID":"c7782764-acee-4624-9eac-97719f137ffd","Type":"ContainerStarted","Data":"61ec32afdb8529681bc27eb0bba495812818c9f916e3fa376a0ce3c04ce44dc3"}
Jul 17 20:39:01.000630 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:01.000595    2572 generic.go:356] "Generic (PLEG): container finished" podID="c7782764-acee-4624-9eac-97719f137ffd" containerID="a6943a60c264aad2d53662d6a3dbdf80acee44718cc6c578dab6bdb2a54f8f35" exitCode=6
Jul 17 20:39:01.001122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:01.000679    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" event={"ID":"c7782764-acee-4624-9eac-97719f137ffd","Type":"ContainerDied","Data":"a6943a60c264aad2d53662d6a3dbdf80acee44718cc6c578dab6bdb2a54f8f35"}
Jul 17 20:39:01.001122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:01.000896    2572 scope.go:117] "RemoveContainer" containerID="a6943a60c264aad2d53662d6a3dbdf80acee44718cc6c578dab6bdb2a54f8f35"
Jul 17 20:39:02.006691 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:02.006597    2572 generic.go:356] "Generic (PLEG): container finished" podID="c7782764-acee-4624-9eac-97719f137ffd" containerID="f51daf25f9a969f64ac6260224b898c78037acd809c38cfd0232638ae5735044" exitCode=6
Jul 17 20:39:02.007157 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:02.006694    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" event={"ID":"c7782764-acee-4624-9eac-97719f137ffd","Type":"ContainerDied","Data":"f51daf25f9a969f64ac6260224b898c78037acd809c38cfd0232638ae5735044"}
Jul 17 20:39:02.007157 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:02.006762    2572 scope.go:117] "RemoveContainer" containerID="a6943a60c264aad2d53662d6a3dbdf80acee44718cc6c578dab6bdb2a54f8f35"
Jul 17 20:39:02.007157 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:02.006946    2572 scope.go:117] "RemoveContainer" containerID="f51daf25f9a969f64ac6260224b898c78037acd809c38cfd0232638ae5735044"
Jul 17 20:39:02.007318 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:02.007175    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp_odh-ai-gateway-infra(c7782764-acee-4624-9eac-97719f137ffd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" podUID="c7782764-acee-4624-9eac-97719f137ffd"
Jul 17 20:39:03.012752 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:03.012726    2572 scope.go:117] "RemoveContainer" containerID="f51daf25f9a969f64ac6260224b898c78037acd809c38cfd0232638ae5735044"
Jul 17 20:39:03.013173 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:03.012907    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp_odh-ai-gateway-infra(c7782764-acee-4624-9eac-97719f137ffd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" podUID="c7782764-acee-4624-9eac-97719f137ffd"
Jul 17 20:39:06.283762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:06.283724    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:39:06.848269 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:06.848241    2572 scope.go:117] "RemoveContainer" containerID="e7a2389867e43a4351e46a7680f1ff346286e51dcc8c150067b75d013d4dcf93"
Jul 17 20:39:07.418620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.418535    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"]
Jul 17 20:39:07.421767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.421750    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"
Jul 17 20:39:07.429067 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.429013    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa952aae-028b-450d-8f24-238872b8f9b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2\" (UID: \"aa952aae-028b-450d-8f24-238872b8f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"
Jul 17 20:39:07.438492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.438463    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"]
Jul 17 20:39:07.529916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.529880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa952aae-028b-450d-8f24-238872b8f9b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2\" (UID: \"aa952aae-028b-450d-8f24-238872b8f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"
Jul 17 20:39:07.530558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.530535    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa952aae-028b-450d-8f24-238872b8f9b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2\" (UID: \"aa952aae-028b-450d-8f24-238872b8f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"
Jul 17 20:39:07.732343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.732303    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"
Jul 17 20:39:07.868569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:07.868450    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"]
Jul 17 20:39:07.871256 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:39:07.871224    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa952aae_028b_450d_8f24_238872b8f9b6.slice/crio-67089cf398958a66319908f9fab713f29099ff9bc3fb58f1602cee4dd9613c7a WatchSource:0}: Error finding container 67089cf398958a66319908f9fab713f29099ff9bc3fb58f1602cee4dd9613c7a: Status 404 returned error can't find the container with id 67089cf398958a66319908f9fab713f29099ff9bc3fb58f1602cee4dd9613c7a
Jul 17 20:39:08.044764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.044737    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" event={"ID":"aa952aae-028b-450d-8f24-238872b8f9b6","Type":"ContainerStarted","Data":"67089cf398958a66319908f9fab713f29099ff9bc3fb58f1602cee4dd9613c7a"}
Jul 17 20:39:08.046471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.046448    2572 generic.go:356] "Generic (PLEG): container finished" podID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerID="2ceecc048ac05c6d7b327433f365892c8dbce6c84e88655c1dfe2bbbbb276606" exitCode=6
Jul 17 20:39:08.046573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.046494    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" event={"ID":"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b","Type":"ContainerDied","Data":"2ceecc048ac05c6d7b327433f365892c8dbce6c84e88655c1dfe2bbbbb276606"}
Jul 17 20:39:08.046573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.046548    2572 scope.go:117] "RemoveContainer" containerID="e7a2389867e43a4351e46a7680f1ff346286e51dcc8c150067b75d013d4dcf93"
Jul 17 20:39:08.046828 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.046809    2572 scope.go:117] "RemoveContainer" containerID="2ceecc048ac05c6d7b327433f365892c8dbce6c84e88655c1dfe2bbbbb276606"
Jul 17 20:39:08.047025 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:08.047007    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-pjsmq_odh-ai-gateway-infra(d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b"
Jul 17 20:39:08.538674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.538644    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"]
Jul 17 20:39:08.541809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.541792    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"
Jul 17 20:39:08.549104 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.549074    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"]
Jul 17 20:39:08.638887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.638844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k\" (UID: \"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"
Jul 17 20:39:08.739873 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.739836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k\" (UID: \"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"
Jul 17 20:39:08.740497 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.740480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k\" (UID: \"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"
Jul 17 20:39:08.899908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:08.899814    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"
Jul 17 20:39:09.052650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.052619    2572 generic.go:356] "Generic (PLEG): container finished" podID="aa952aae-028b-450d-8f24-238872b8f9b6" containerID="13105b1fc1b77658095c135985dd599b40e3bf9dcdcbf9daaf404e54fefbd372" exitCode=6
Jul 17 20:39:09.052855 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.052705    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" event={"ID":"aa952aae-028b-450d-8f24-238872b8f9b6","Type":"ContainerDied","Data":"13105b1fc1b77658095c135985dd599b40e3bf9dcdcbf9daaf404e54fefbd372"}
Jul 17 20:39:09.052948 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.052896    2572 scope.go:117] "RemoveContainer" containerID="13105b1fc1b77658095c135985dd599b40e3bf9dcdcbf9daaf404e54fefbd372"
Jul 17 20:39:09.056755 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.056739    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"]
Jul 17 20:39:09.059339 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:39:09.059319    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2fd047_a1e1_400a_83ac_78d5bc5b3d7f.slice/crio-296e6be36caf7df8b1c496be62dc9fbf5035ea046f591995c9589d1ac96d7acb WatchSource:0}: Error finding container 296e6be36caf7df8b1c496be62dc9fbf5035ea046f591995c9589d1ac96d7acb: Status 404 returned error can't find the container with id 296e6be36caf7df8b1c496be62dc9fbf5035ea046f591995c9589d1ac96d7acb
Jul 17 20:39:09.077431 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.077407    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"]
Jul 17 20:39:09.202020 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.201994    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"
Jul 17 20:39:09.243189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.243165    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b-maas-api-service-ca\") pod \"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b\" (UID: \"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b\") "
Jul 17 20:39:09.243511 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.243491    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" (UID: "d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:39:09.343651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:09.343628    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:39:10.061974 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.061877    2572 generic.go:356] "Generic (PLEG): container finished" podID="aa952aae-028b-450d-8f24-238872b8f9b6" containerID="e9a9a4ab5cd7cd1a3c4b41c2f2645ad2a1f87f2e21734c0aa601ed04e609c40e" exitCode=6
Jul 17 20:39:10.062445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.061962    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" event={"ID":"aa952aae-028b-450d-8f24-238872b8f9b6","Type":"ContainerDied","Data":"e9a9a4ab5cd7cd1a3c4b41c2f2645ad2a1f87f2e21734c0aa601ed04e609c40e"}
Jul 17 20:39:10.062445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.062035    2572 scope.go:117] "RemoveContainer" containerID="13105b1fc1b77658095c135985dd599b40e3bf9dcdcbf9daaf404e54fefbd372"
Jul 17 20:39:10.062445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.062301    2572 scope.go:117] "RemoveContainer" containerID="e9a9a4ab5cd7cd1a3c4b41c2f2645ad2a1f87f2e21734c0aa601ed04e609c40e"
Jul 17 20:39:10.062567 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:10.062545    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2_odh-ai-gateway-infra(aa952aae-028b-450d-8f24-238872b8f9b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" podUID="aa952aae-028b-450d-8f24-238872b8f9b6"
Jul 17 20:39:10.063839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.063814    2572 generic.go:356] "Generic (PLEG): container finished" podID="ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f" containerID="52bed5fa0aa1388b64a056b6a3facad1aed0c5526d23ea8046b7934b09070ed8" exitCode=0
Jul 17 20:39:10.063930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.063879    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k" event={"ID":"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f","Type":"ContainerDied","Data":"52bed5fa0aa1388b64a056b6a3facad1aed0c5526d23ea8046b7934b09070ed8"}
Jul 17 20:39:10.063930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.063922    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k" event={"ID":"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f","Type":"ContainerStarted","Data":"296e6be36caf7df8b1c496be62dc9fbf5035ea046f591995c9589d1ac96d7acb"}
Jul 17 20:39:10.064999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.064980    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq" event={"ID":"d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b","Type":"ContainerDied","Data":"13b251cee1e36d50b0c30754b389a853d80f7caeed9b4f1e259ad72b9a16a7d2"}
Jul 17 20:39:10.064999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.064998    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"
Jul 17 20:39:10.072961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.072935    2572 scope.go:117] "RemoveContainer" containerID="2ceecc048ac05c6d7b327433f365892c8dbce6c84e88655c1dfe2bbbbb276606"
Jul 17 20:39:10.096727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.096699    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"]
Jul 17 20:39:10.102423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:10.102393    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-pjsmq"]
Jul 17 20:39:11.070409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:11.070380    2572 scope.go:117] "RemoveContainer" containerID="e9a9a4ab5cd7cd1a3c4b41c2f2645ad2a1f87f2e21734c0aa601ed04e609c40e"
Jul 17 20:39:11.070851 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:11.070613    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2_odh-ai-gateway-infra(aa952aae-028b-450d-8f24-238872b8f9b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" podUID="aa952aae-028b-450d-8f24-238872b8f9b6"
Jul 17 20:39:11.200963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:11.200937    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"
Jul 17 20:39:11.257342 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:11.257307    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f-maas-api-service-ca\") pod \"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f\" (UID: \"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f\") "
Jul 17 20:39:11.257699 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:11.257672    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f" (UID: "ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:39:11.357853 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:11.357758    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:39:11.853843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:11.853808    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" path="/var/lib/kubelet/pods/d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b/volumes"
Jul 17 20:39:12.076808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:12.076776    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k" event={"ID":"ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f","Type":"ContainerDied","Data":"296e6be36caf7df8b1c496be62dc9fbf5035ea046f591995c9589d1ac96d7acb"}
Jul 17 20:39:12.076808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:12.076800    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k"
Jul 17 20:39:12.077355 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:12.076823    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="296e6be36caf7df8b1c496be62dc9fbf5035ea046f591995c9589d1ac96d7acb"
Jul 17 20:39:15.220994 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.220961    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"]
Jul 17 20:39:15.221389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221305    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f" containerName="revoke-keys"
Jul 17 20:39:15.221389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221317    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f" containerName="revoke-keys"
Jul 17 20:39:15.221389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221330    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:15.221389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221335    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:15.221389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221352    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:15.221389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221358    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:15.221591 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221403    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f" containerName="revoke-keys"
Jul 17 20:39:15.221591 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221411    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:15.221591 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.221419    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:15.224188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.224169    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"
Jul 17 20:39:15.234538 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.234506    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"]
Jul 17 20:39:15.284391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.284353    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a937-0241-4de0-99bf-7d06e15e7e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg\" (UID: \"0af6a937-0241-4de0-99bf-7d06e15e7e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"
Jul 17 20:39:15.385640 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.385605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a937-0241-4de0-99bf-7d06e15e7e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg\" (UID: \"0af6a937-0241-4de0-99bf-7d06e15e7e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"
Jul 17 20:39:15.386246 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.386229    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a937-0241-4de0-99bf-7d06e15e7e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg\" (UID: \"0af6a937-0241-4de0-99bf-7d06e15e7e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"
Jul 17 20:39:15.534191 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.534113    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"
Jul 17 20:39:15.662032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:15.662001    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"]
Jul 17 20:39:15.664844 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:39:15.664817    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af6a937_0241_4de0_99bf_7d06e15e7e6d.slice/crio-267526e3ef4dadd7b45487d8e4c58357b9e8fc27a0d13f64774b02658a097223 WatchSource:0}: Error finding container 267526e3ef4dadd7b45487d8e4c58357b9e8fc27a0d13f64774b02658a097223: Status 404 returned error can't find the container with id 267526e3ef4dadd7b45487d8e4c58357b9e8fc27a0d13f64774b02658a097223
Jul 17 20:39:16.095388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:16.095303    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" event={"ID":"0af6a937-0241-4de0-99bf-7d06e15e7e6d","Type":"ContainerStarted","Data":"cc59e32c3d7dce6fba5a1fe420997933d1d83a8df36fe8bf56e57e7911016e41"}
Jul 17 20:39:16.095388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:16.095354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" event={"ID":"0af6a937-0241-4de0-99bf-7d06e15e7e6d","Type":"ContainerStarted","Data":"267526e3ef4dadd7b45487d8e4c58357b9e8fc27a0d13f64774b02658a097223"}
Jul 17 20:39:16.111416 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:16.111366    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" podStartSLOduration=1.111350071 podStartE2EDuration="1.111350071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:39:16.110149057 +0000 UTC m=+2590.804125199" watchObservedRunningTime="2026-07-17 20:39:16.111350071 +0000 UTC m=+2590.805326216"
Jul 17 20:39:16.848839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:16.848810    2572 scope.go:117] "RemoveContainer" containerID="f51daf25f9a969f64ac6260224b898c78037acd809c38cfd0232638ae5735044"
Jul 17 20:39:17.101151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:17.101036    2572 generic.go:356] "Generic (PLEG): container finished" podID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerID="cc59e32c3d7dce6fba5a1fe420997933d1d83a8df36fe8bf56e57e7911016e41" exitCode=6
Jul 17 20:39:17.101151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:17.101116    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" event={"ID":"0af6a937-0241-4de0-99bf-7d06e15e7e6d","Type":"ContainerDied","Data":"cc59e32c3d7dce6fba5a1fe420997933d1d83a8df36fe8bf56e57e7911016e41"}
Jul 17 20:39:17.101353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:17.101337    2572 scope.go:117] "RemoveContainer" containerID="cc59e32c3d7dce6fba5a1fe420997933d1d83a8df36fe8bf56e57e7911016e41"
Jul 17 20:39:17.102864 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:17.102845    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" event={"ID":"c7782764-acee-4624-9eac-97719f137ffd","Type":"ContainerStarted","Data":"204688753e1592f1bfe1fddc526db660e593a64beb3ac0192a6c6bcbd03b687d"}
Jul 17 20:39:17.134948 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:17.134902    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" podStartSLOduration=18.134888899 podStartE2EDuration="18.134888899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:39:17.133520707 +0000 UTC m=+2591.827496853" watchObservedRunningTime="2026-07-17 20:39:17.134888899 +0000 UTC m=+2591.828865044"
Jul 17 20:39:18.109114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.109020    2572 generic.go:356] "Generic (PLEG): container finished" podID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerID="2ff0da87e8a88ac052ef650f98d63b47280b7ec2dbbeec4fb71ca1cbcaebd14c" exitCode=6
Jul 17 20:39:18.109513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.109139    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" event={"ID":"0af6a937-0241-4de0-99bf-7d06e15e7e6d","Type":"ContainerDied","Data":"2ff0da87e8a88ac052ef650f98d63b47280b7ec2dbbeec4fb71ca1cbcaebd14c"}
Jul 17 20:39:18.109513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.109199    2572 scope.go:117] "RemoveContainer" containerID="cc59e32c3d7dce6fba5a1fe420997933d1d83a8df36fe8bf56e57e7911016e41"
Jul 17 20:39:18.109513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.109292    2572 scope.go:117] "RemoveContainer" containerID="2ff0da87e8a88ac052ef650f98d63b47280b7ec2dbbeec4fb71ca1cbcaebd14c"
Jul 17 20:39:18.109513 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:18.109502    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg_odh-ai-gateway-infra(0af6a937-0241-4de0-99bf-7d06e15e7e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d"
Jul 17 20:39:18.111064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.111027    2572 generic.go:356] "Generic (PLEG): container finished" podID="c7782764-acee-4624-9eac-97719f137ffd" containerID="204688753e1592f1bfe1fddc526db660e593a64beb3ac0192a6c6bcbd03b687d" exitCode=6
Jul 17 20:39:18.111165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.111064    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" event={"ID":"c7782764-acee-4624-9eac-97719f137ffd","Type":"ContainerDied","Data":"204688753e1592f1bfe1fddc526db660e593a64beb3ac0192a6c6bcbd03b687d"}
Jul 17 20:39:18.111296 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.111281    2572 scope.go:117] "RemoveContainer" containerID="204688753e1592f1bfe1fddc526db660e593a64beb3ac0192a6c6bcbd03b687d"
Jul 17 20:39:18.111439 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:18.111424    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp_odh-ai-gateway-infra(c7782764-acee-4624-9eac-97719f137ffd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" podUID="c7782764-acee-4624-9eac-97719f137ffd"
Jul 17 20:39:18.119778 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.119762    2572 scope.go:117] "RemoveContainer" containerID="f51daf25f9a969f64ac6260224b898c78037acd809c38cfd0232638ae5735044"
Jul 17 20:39:18.144585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:18.144553    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"]
Jul 17 20:39:19.117376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:19.117350    2572 scope.go:117] "RemoveContainer" containerID="2ff0da87e8a88ac052ef650f98d63b47280b7ec2dbbeec4fb71ca1cbcaebd14c"
Jul 17 20:39:19.117816 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:19.117525    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg_odh-ai-gateway-infra(0af6a937-0241-4de0-99bf-7d06e15e7e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d"
Jul 17 20:39:19.249411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:19.249381    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"
Jul 17 20:39:19.317007 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:19.316978    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7782764-acee-4624-9eac-97719f137ffd-maas-api-service-ca\") pod \"c7782764-acee-4624-9eac-97719f137ffd\" (UID: \"c7782764-acee-4624-9eac-97719f137ffd\") "
Jul 17 20:39:19.317354 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:19.317331    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7782764-acee-4624-9eac-97719f137ffd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7782764-acee-4624-9eac-97719f137ffd" (UID: "c7782764-acee-4624-9eac-97719f137ffd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:39:19.418409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:19.418330    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7782764-acee-4624-9eac-97719f137ffd-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:39:20.124420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:20.124331    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"
Jul 17 20:39:20.124420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:20.124345    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp" event={"ID":"c7782764-acee-4624-9eac-97719f137ffd","Type":"ContainerDied","Data":"61ec32afdb8529681bc27eb0bba495812818c9f916e3fa376a0ce3c04ce44dc3"}
Jul 17 20:39:20.124420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:20.124407    2572 scope.go:117] "RemoveContainer" containerID="204688753e1592f1bfe1fddc526db660e593a64beb3ac0192a6c6bcbd03b687d"
Jul 17 20:39:20.140929 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:20.140902    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"]
Jul 17 20:39:20.144487 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:20.144464    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-wx8vp"]
Jul 17 20:39:21.853072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:21.853009    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7782764-acee-4624-9eac-97719f137ffd" path="/var/lib/kubelet/pods/c7782764-acee-4624-9eac-97719f137ffd/volumes"
Jul 17 20:39:22.477682 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:22.477644    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:39:23.848632 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:23.848595    2572 scope.go:117] "RemoveContainer" containerID="e9a9a4ab5cd7cd1a3c4b41c2f2645ad2a1f87f2e21734c0aa601ed04e609c40e"
Jul 17 20:39:24.145212 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:24.145185    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" event={"ID":"aa952aae-028b-450d-8f24-238872b8f9b6","Type":"ContainerStarted","Data":"8fa5e576d73fa1b70f1daab21ce9d4450f791f9753f68a0ca2c9722b15016a89"}
Jul 17 20:39:24.162822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:24.162776    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" podStartSLOduration=17.162761194 podStartE2EDuration="17.162761194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:39:24.161522565 +0000 UTC m=+2598.855498708" watchObservedRunningTime="2026-07-17 20:39:24.162761194 +0000 UTC m=+2598.856737341"
Jul 17 20:39:25.151736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:25.151157    2572 generic.go:356] "Generic (PLEG): container finished" podID="aa952aae-028b-450d-8f24-238872b8f9b6" containerID="8fa5e576d73fa1b70f1daab21ce9d4450f791f9753f68a0ca2c9722b15016a89" exitCode=6
Jul 17 20:39:25.151736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:25.151235    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" event={"ID":"aa952aae-028b-450d-8f24-238872b8f9b6","Type":"ContainerDied","Data":"8fa5e576d73fa1b70f1daab21ce9d4450f791f9753f68a0ca2c9722b15016a89"}
Jul 17 20:39:25.151736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:25.151287    2572 scope.go:117] "RemoveContainer" containerID="e9a9a4ab5cd7cd1a3c4b41c2f2645ad2a1f87f2e21734c0aa601ed04e609c40e"
Jul 17 20:39:25.151736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:25.151642    2572 scope.go:117] "RemoveContainer" containerID="8fa5e576d73fa1b70f1daab21ce9d4450f791f9753f68a0ca2c9722b15016a89"
Jul 17 20:39:25.152336 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:25.151858    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2_odh-ai-gateway-infra(aa952aae-028b-450d-8f24-238872b8f9b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" podUID="aa952aae-028b-450d-8f24-238872b8f9b6"
Jul 17 20:39:25.171618 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:25.171596    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"]
Jul 17 20:39:26.289838 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:26.289815    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"
Jul 17 20:39:26.373977 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:26.373938    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa952aae-028b-450d-8f24-238872b8f9b6-maas-api-service-ca\") pod \"aa952aae-028b-450d-8f24-238872b8f9b6\" (UID: \"aa952aae-028b-450d-8f24-238872b8f9b6\") "
Jul 17 20:39:26.374325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:26.374302    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa952aae-028b-450d-8f24-238872b8f9b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa952aae-028b-450d-8f24-238872b8f9b6" (UID: "aa952aae-028b-450d-8f24-238872b8f9b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:39:26.474984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:26.474958    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa952aae-028b-450d-8f24-238872b8f9b6-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:39:27.168265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:27.168232    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"
Jul 17 20:39:27.168449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:27.168228    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2" event={"ID":"aa952aae-028b-450d-8f24-238872b8f9b6","Type":"ContainerDied","Data":"67089cf398958a66319908f9fab713f29099ff9bc3fb58f1602cee4dd9613c7a"}
Jul 17 20:39:27.168449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:27.168371    2572 scope.go:117] "RemoveContainer" containerID="8fa5e576d73fa1b70f1daab21ce9d4450f791f9753f68a0ca2c9722b15016a89"
Jul 17 20:39:27.190103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:27.190077    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"]
Jul 17 20:39:27.195167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:27.195145    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-gjmj2"]
Jul 17 20:39:27.853295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:27.853262    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" path="/var/lib/kubelet/pods/aa952aae-028b-450d-8f24-238872b8f9b6/volumes"
Jul 17 20:39:32.848356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:32.848323    2572 scope.go:117] "RemoveContainer" containerID="2ff0da87e8a88ac052ef650f98d63b47280b7ec2dbbeec4fb71ca1cbcaebd14c"
Jul 17 20:39:33.196310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:33.196278    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" event={"ID":"0af6a937-0241-4de0-99bf-7d06e15e7e6d","Type":"ContainerStarted","Data":"1a5a102c0951e52accf7519b08df21830896c0359db9f94b0bb5560d8e738354"}
Jul 17 20:39:34.202548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:34.202504    2572 generic.go:356] "Generic (PLEG): container finished" podID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerID="1a5a102c0951e52accf7519b08df21830896c0359db9f94b0bb5560d8e738354" exitCode=6
Jul 17 20:39:34.202964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:34.202575    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" event={"ID":"0af6a937-0241-4de0-99bf-7d06e15e7e6d","Type":"ContainerDied","Data":"1a5a102c0951e52accf7519b08df21830896c0359db9f94b0bb5560d8e738354"}
Jul 17 20:39:34.202964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:34.202641    2572 scope.go:117] "RemoveContainer" containerID="2ff0da87e8a88ac052ef650f98d63b47280b7ec2dbbeec4fb71ca1cbcaebd14c"
Jul 17 20:39:34.202964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:34.202910    2572 scope.go:117] "RemoveContainer" containerID="1a5a102c0951e52accf7519b08df21830896c0359db9f94b0bb5560d8e738354"
Jul 17 20:39:34.203185 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:34.203156    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg_odh-ai-gateway-infra(0af6a937-0241-4de0-99bf-7d06e15e7e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d"
Jul 17 20:39:34.223367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:34.223343    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"]
Jul 17 20:39:35.345191 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:35.345166    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"
Jul 17 20:39:35.446038 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:35.446008    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a937-0241-4de0-99bf-7d06e15e7e6d-maas-api-service-ca\") pod \"0af6a937-0241-4de0-99bf-7d06e15e7e6d\" (UID: \"0af6a937-0241-4de0-99bf-7d06e15e7e6d\") "
Jul 17 20:39:35.446374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:35.446351    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af6a937-0241-4de0-99bf-7d06e15e7e6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0af6a937-0241-4de0-99bf-7d06e15e7e6d" (UID: "0af6a937-0241-4de0-99bf-7d06e15e7e6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:39:35.546740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:35.546708    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a937-0241-4de0-99bf-7d06e15e7e6d-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:39:36.215153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:36.215117    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg" event={"ID":"0af6a937-0241-4de0-99bf-7d06e15e7e6d","Type":"ContainerDied","Data":"267526e3ef4dadd7b45487d8e4c58357b9e8fc27a0d13f64774b02658a097223"}
Jul 17 20:39:36.215153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:36.215149    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"
Jul 17 20:39:36.215402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:36.215182    2572 scope.go:117] "RemoveContainer" containerID="1a5a102c0951e52accf7519b08df21830896c0359db9f94b0bb5560d8e738354"
Jul 17 20:39:36.232107 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:36.232077    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"]
Jul 17 20:39:36.240961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:36.240936    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-l4pvg"]
Jul 17 20:39:37.853160 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:37.853122    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" path="/var/lib/kubelet/pods/0af6a937-0241-4de0-99bf-7d06e15e7e6d/volumes"
Jul 17 20:39:41.245269 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245240    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"]
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245561    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245574    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245582    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245587    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245597    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245602    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245607    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245612    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245626    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245637    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245642    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245650    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245656    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245662    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245667    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245713    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245722    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245727    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa952aae-028b-450d-8f24-238872b8f9b6" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245732    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245736    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245741    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a8de07-8b18-4c6f-9ca8-6a9f858cd82b" containerName="revoke-keys"
Jul 17 20:39:41.245727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245746    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:41.246910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.245751    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:41.250194 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.250172    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"
Jul 17 20:39:41.252809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.252782    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:39:41.253070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.253038    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:39:41.258708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.258683    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"]
Jul 17 20:39:41.291329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.291304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906fd34c-5f61-43be-b273-22bae82fcbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-bx9nc\" (UID: \"906fd34c-5f61-43be-b273-22bae82fcbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"
Jul 17 20:39:41.392224 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.392191    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906fd34c-5f61-43be-b273-22bae82fcbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-bx9nc\" (UID: \"906fd34c-5f61-43be-b273-22bae82fcbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"
Jul 17 20:39:41.392836 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.392817    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906fd34c-5f61-43be-b273-22bae82fcbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-bx9nc\" (UID: \"906fd34c-5f61-43be-b273-22bae82fcbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"
Jul 17 20:39:41.561555 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.561466    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"
Jul 17 20:39:41.690726 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:41.690705    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"]
Jul 17 20:39:41.692990 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:39:41.692965    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906fd34c_5f61_43be_b273_22bae82fcbb8.slice/crio-d1d91288b25f10fab97f870c704b9f6d7cc5427607617ab1cd532e97e167f1d8 WatchSource:0}: Error finding container d1d91288b25f10fab97f870c704b9f6d7cc5427607617ab1cd532e97e167f1d8: Status 404 returned error can't find the container with id d1d91288b25f10fab97f870c704b9f6d7cc5427607617ab1cd532e97e167f1d8
Jul 17 20:39:42.247364 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:42.247329    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" event={"ID":"906fd34c-5f61-43be-b273-22bae82fcbb8","Type":"ContainerStarted","Data":"16c5a3678fd8a4297d6b2e3a0f887a59acd03828dd8bbc8f98517d9f4ad51f0e"}
Jul 17 20:39:42.247716 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:42.247392    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" event={"ID":"906fd34c-5f61-43be-b273-22bae82fcbb8","Type":"ContainerStarted","Data":"d1d91288b25f10fab97f870c704b9f6d7cc5427607617ab1cd532e97e167f1d8"}
Jul 17 20:39:42.261640 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:42.261593    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" podStartSLOduration=1.261575737 podStartE2EDuration="1.261575737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:39:42.261171976 +0000 UTC m=+2616.955148123" watchObservedRunningTime="2026-07-17 20:39:42.261575737 +0000 UTC m=+2616.955551885"
Jul 17 20:39:43.253546 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:43.253508    2572 generic.go:356] "Generic (PLEG): container finished" podID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerID="16c5a3678fd8a4297d6b2e3a0f887a59acd03828dd8bbc8f98517d9f4ad51f0e" exitCode=6
Jul 17 20:39:43.253953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:43.253601    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" event={"ID":"906fd34c-5f61-43be-b273-22bae82fcbb8","Type":"ContainerDied","Data":"16c5a3678fd8a4297d6b2e3a0f887a59acd03828dd8bbc8f98517d9f4ad51f0e"}
Jul 17 20:39:43.253953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:43.253764    2572 scope.go:117] "RemoveContainer" containerID="16c5a3678fd8a4297d6b2e3a0f887a59acd03828dd8bbc8f98517d9f4ad51f0e"
Jul 17 20:39:44.259470 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:44.259435    2572 generic.go:356] "Generic (PLEG): container finished" podID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerID="aa72a970a78411588800341105d44c8cb237fb1d4f0f3822195b63bf11145b2a" exitCode=6
Jul 17 20:39:44.259910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:44.259476    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" event={"ID":"906fd34c-5f61-43be-b273-22bae82fcbb8","Type":"ContainerDied","Data":"aa72a970a78411588800341105d44c8cb237fb1d4f0f3822195b63bf11145b2a"}
Jul 17 20:39:44.259910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:44.259524    2572 scope.go:117] "RemoveContainer" containerID="16c5a3678fd8a4297d6b2e3a0f887a59acd03828dd8bbc8f98517d9f4ad51f0e"
Jul 17 20:39:44.259910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:44.259676    2572 scope.go:117] "RemoveContainer" containerID="aa72a970a78411588800341105d44c8cb237fb1d4f0f3822195b63bf11145b2a"
Jul 17 20:39:44.259910 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:44.259869    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-bx9nc_odh-ai-gateway-infra(906fd34c-5f61-43be-b273-22bae82fcbb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8"
Jul 17 20:39:45.265701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:45.265676    2572 scope.go:117] "RemoveContainer" containerID="aa72a970a78411588800341105d44c8cb237fb1d4f0f3822195b63bf11145b2a"
Jul 17 20:39:45.266084 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:39:45.265858    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-bx9nc_odh-ai-gateway-infra(906fd34c-5f61-43be-b273-22bae82fcbb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8"
Jul 17 20:39:57.458728 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.458694    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"]
Jul 17 20:39:57.459119 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.459027    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:57.459119 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.459042    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:57.459119 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.459075    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:57.459119 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.459081    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:57.459255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.459138    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af6a937-0241-4de0-99bf-7d06e15e7e6d" containerName="revoke-keys"
Jul 17 20:39:57.459255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.459149    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7782764-acee-4624-9eac-97719f137ffd" containerName="revoke-keys"
Jul 17 20:39:57.469681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.469647    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"
Jul 17 20:39:57.470773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.470743    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"]
Jul 17 20:39:57.623846 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.623806    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07d87a-6ba8-4142-88b2-79b9b37a99f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw\" (UID: \"ef07d87a-6ba8-4142-88b2-79b9b37a99f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"
Jul 17 20:39:57.724928 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.724890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07d87a-6ba8-4142-88b2-79b9b37a99f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw\" (UID: \"ef07d87a-6ba8-4142-88b2-79b9b37a99f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"
Jul 17 20:39:57.725550 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.725530    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07d87a-6ba8-4142-88b2-79b9b37a99f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw\" (UID: \"ef07d87a-6ba8-4142-88b2-79b9b37a99f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"
Jul 17 20:39:57.780970 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.780946    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"
Jul 17 20:39:57.917447 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:57.917420    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"]
Jul 17 20:39:57.920073 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:39:57.920031    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef07d87a_6ba8_4142_88b2_79b9b37a99f3.slice/crio-2c85b2242da8604b9ef9ce8c688428a6a840244464d760335eadee18e25cf9ce WatchSource:0}: Error finding container 2c85b2242da8604b9ef9ce8c688428a6a840244464d760335eadee18e25cf9ce: Status 404 returned error can't find the container with id 2c85b2242da8604b9ef9ce8c688428a6a840244464d760335eadee18e25cf9ce
Jul 17 20:39:58.324329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:58.324288    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" event={"ID":"ef07d87a-6ba8-4142-88b2-79b9b37a99f3","Type":"ContainerStarted","Data":"fdcffb2692a6066a1c87947bb8bbeecd73d12b089f682e483f8080d1b3ed3e86"}
Jul 17 20:39:58.324508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:58.324355    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" event={"ID":"ef07d87a-6ba8-4142-88b2-79b9b37a99f3","Type":"ContainerStarted","Data":"2c85b2242da8604b9ef9ce8c688428a6a840244464d760335eadee18e25cf9ce"}
Jul 17 20:39:58.340355 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:58.340307    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" podStartSLOduration=1.3402920360000001 podStartE2EDuration="1.340292036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:39:58.338756005 +0000 UTC m=+2633.032732153" watchObservedRunningTime="2026-07-17 20:39:58.340292036 +0000 UTC m=+2633.034268159"
Jul 17 20:39:58.847953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:58.847930    2572 scope.go:117] "RemoveContainer" containerID="aa72a970a78411588800341105d44c8cb237fb1d4f0f3822195b63bf11145b2a"
Jul 17 20:39:59.331619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.331530    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerID="fdcffb2692a6066a1c87947bb8bbeecd73d12b089f682e483f8080d1b3ed3e86" exitCode=6
Jul 17 20:39:59.331771 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.331619    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" event={"ID":"ef07d87a-6ba8-4142-88b2-79b9b37a99f3","Type":"ContainerDied","Data":"fdcffb2692a6066a1c87947bb8bbeecd73d12b089f682e483f8080d1b3ed3e86"}
Jul 17 20:39:59.331831 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.331790    2572 scope.go:117] "RemoveContainer" containerID="fdcffb2692a6066a1c87947bb8bbeecd73d12b089f682e483f8080d1b3ed3e86"
Jul 17 20:39:59.333705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.333686    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" event={"ID":"906fd34c-5f61-43be-b273-22bae82fcbb8","Type":"ContainerStarted","Data":"50ffc797cd9c0a01548126834141e45430a44963d1166a730f498f4ffbf87c8e"}
Jul 17 20:39:59.336603 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.336584    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"]
Jul 17 20:39:59.340421 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.340403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"
Jul 17 20:39:59.348094 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.348071    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"]
Jul 17 20:39:59.437500 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.437407    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b94e51-ea1f-49e9-bd1f-7e18dbce535f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq\" (UID: \"78b94e51-ea1f-49e9-bd1f-7e18dbce535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"
Jul 17 20:39:59.538341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.538308    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b94e51-ea1f-49e9-bd1f-7e18dbce535f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq\" (UID: \"78b94e51-ea1f-49e9-bd1f-7e18dbce535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"
Jul 17 20:39:59.538922 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.538904    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b94e51-ea1f-49e9-bd1f-7e18dbce535f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq\" (UID: \"78b94e51-ea1f-49e9-bd1f-7e18dbce535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"
Jul 17 20:39:59.670835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.670804    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"
Jul 17 20:39:59.798601 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:39:59.798575    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"]
Jul 17 20:39:59.800632 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:39:59.800608    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b94e51_ea1f_49e9_bd1f_7e18dbce535f.slice/crio-eaf40dbe00d6e152d6f32b666666c40ae8991c514665967a61d947c0a1ba23d1 WatchSource:0}: Error finding container eaf40dbe00d6e152d6f32b666666c40ae8991c514665967a61d947c0a1ba23d1: Status 404 returned error can't find the container with id eaf40dbe00d6e152d6f32b666666c40ae8991c514665967a61d947c0a1ba23d1
Jul 17 20:40:00.340005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.339911    2572 generic.go:356] "Generic (PLEG): container finished" podID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerID="50ffc797cd9c0a01548126834141e45430a44963d1166a730f498f4ffbf87c8e" exitCode=6
Jul 17 20:40:00.340005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.339986    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" event={"ID":"906fd34c-5f61-43be-b273-22bae82fcbb8","Type":"ContainerDied","Data":"50ffc797cd9c0a01548126834141e45430a44963d1166a730f498f4ffbf87c8e"}
Jul 17 20:40:00.340517 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.340066    2572 scope.go:117] "RemoveContainer" containerID="aa72a970a78411588800341105d44c8cb237fb1d4f0f3822195b63bf11145b2a"
Jul 17 20:40:00.340517 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.340347    2572 scope.go:117] "RemoveContainer" containerID="50ffc797cd9c0a01548126834141e45430a44963d1166a730f498f4ffbf87c8e"
Jul 17 20:40:00.340630 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:00.340562    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-bx9nc_odh-ai-gateway-infra(906fd34c-5f61-43be-b273-22bae82fcbb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8"
Jul 17 20:40:00.342124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.342100    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerID="01d69e2ce823ebc3e5451f6f4937759f1f62ed948e010a1723efaf8c3ecddf68" exitCode=6
Jul 17 20:40:00.342275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.342186    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" event={"ID":"ef07d87a-6ba8-4142-88b2-79b9b37a99f3","Type":"ContainerDied","Data":"01d69e2ce823ebc3e5451f6f4937759f1f62ed948e010a1723efaf8c3ecddf68"}
Jul 17 20:40:00.342433 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.342406    2572 scope.go:117] "RemoveContainer" containerID="01d69e2ce823ebc3e5451f6f4937759f1f62ed948e010a1723efaf8c3ecddf68"
Jul 17 20:40:00.342605 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:00.342585    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw_odh-ai-gateway-infra(ef07d87a-6ba8-4142-88b2-79b9b37a99f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3"
Jul 17 20:40:00.343702 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.343665    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" event={"ID":"78b94e51-ea1f-49e9-bd1f-7e18dbce535f","Type":"ContainerStarted","Data":"8a56901738d9e23e6ab072e9033257543d1dee4c2d86f030e0ebea3ce98e82fd"}
Jul 17 20:40:00.343815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.343714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" event={"ID":"78b94e51-ea1f-49e9-bd1f-7e18dbce535f","Type":"ContainerStarted","Data":"eaf40dbe00d6e152d6f32b666666c40ae8991c514665967a61d947c0a1ba23d1"}
Jul 17 20:40:00.350991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.350971    2572 scope.go:117] "RemoveContainer" containerID="fdcffb2692a6066a1c87947bb8bbeecd73d12b089f682e483f8080d1b3ed3e86"
Jul 17 20:40:00.376736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.376717    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"]
Jul 17 20:40:00.387575 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:00.387530    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" podStartSLOduration=1.387518224 podStartE2EDuration="1.387518224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:40:00.385830518 +0000 UTC m=+2635.079806677" watchObservedRunningTime="2026-07-17 20:40:00.387518224 +0000 UTC m=+2635.081494369"
Jul 17 20:40:01.349568 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.349533    2572 generic.go:356] "Generic (PLEG): container finished" podID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerID="8a56901738d9e23e6ab072e9033257543d1dee4c2d86f030e0ebea3ce98e82fd" exitCode=6
Jul 17 20:40:01.350030 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.349567    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" event={"ID":"78b94e51-ea1f-49e9-bd1f-7e18dbce535f","Type":"ContainerDied","Data":"8a56901738d9e23e6ab072e9033257543d1dee4c2d86f030e0ebea3ce98e82fd"}
Jul 17 20:40:01.350030 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.349797    2572 scope.go:117] "RemoveContainer" containerID="8a56901738d9e23e6ab072e9033257543d1dee4c2d86f030e0ebea3ce98e82fd"
Jul 17 20:40:01.353006 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.352990    2572 scope.go:117] "RemoveContainer" containerID="01d69e2ce823ebc3e5451f6f4937759f1f62ed948e010a1723efaf8c3ecddf68"
Jul 17 20:40:01.353267 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:01.353190    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw_odh-ai-gateway-infra(ef07d87a-6ba8-4142-88b2-79b9b37a99f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3"
Jul 17 20:40:01.487034 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.487006    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"
Jul 17 20:40:01.553915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.553886    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906fd34c-5f61-43be-b273-22bae82fcbb8-maas-api-service-ca\") pod \"906fd34c-5f61-43be-b273-22bae82fcbb8\" (UID: \"906fd34c-5f61-43be-b273-22bae82fcbb8\") "
Jul 17 20:40:01.554271 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.554247    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906fd34c-5f61-43be-b273-22bae82fcbb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "906fd34c-5f61-43be-b273-22bae82fcbb8" (UID: "906fd34c-5f61-43be-b273-22bae82fcbb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:40:01.655166 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:01.655141    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906fd34c-5f61-43be-b273-22bae82fcbb8-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:40:02.359468 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.359434    2572 generic.go:356] "Generic (PLEG): container finished" podID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerID="bf0b86adec87b5653809ff19716bdca9c11ce6a47031dd9cd02c50d2476b1f9e" exitCode=6
Jul 17 20:40:02.359881 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.359514    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" event={"ID":"78b94e51-ea1f-49e9-bd1f-7e18dbce535f","Type":"ContainerDied","Data":"bf0b86adec87b5653809ff19716bdca9c11ce6a47031dd9cd02c50d2476b1f9e"}
Jul 17 20:40:02.359881 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.359597    2572 scope.go:117] "RemoveContainer" containerID="8a56901738d9e23e6ab072e9033257543d1dee4c2d86f030e0ebea3ce98e82fd"
Jul 17 20:40:02.359881 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.359809    2572 scope.go:117] "RemoveContainer" containerID="bf0b86adec87b5653809ff19716bdca9c11ce6a47031dd9cd02c50d2476b1f9e"
Jul 17 20:40:02.360026 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:02.360014    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq_odh-ai-gateway-infra(78b94e51-ea1f-49e9-bd1f-7e18dbce535f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f"
Jul 17 20:40:02.360916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.360893    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc" event={"ID":"906fd34c-5f61-43be-b273-22bae82fcbb8","Type":"ContainerDied","Data":"d1d91288b25f10fab97f870c704b9f6d7cc5427607617ab1cd532e97e167f1d8"}
Jul 17 20:40:02.360996 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.360907    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"
Jul 17 20:40:02.370004 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.369986    2572 scope.go:117] "RemoveContainer" containerID="50ffc797cd9c0a01548126834141e45430a44963d1166a730f498f4ffbf87c8e"
Jul 17 20:40:02.388774 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.388754    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"]
Jul 17 20:40:02.392400 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.392381    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-bx9nc"]
Jul 17 20:40:02.482492 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:02.482462    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:40:03.367747 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:03.367720    2572 scope.go:117] "RemoveContainer" containerID="bf0b86adec87b5653809ff19716bdca9c11ce6a47031dd9cd02c50d2476b1f9e"
Jul 17 20:40:03.368115 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:03.367908    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq_odh-ai-gateway-infra(78b94e51-ea1f-49e9-bd1f-7e18dbce535f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f"
Jul 17 20:40:03.857866 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:03.857830    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" path="/var/lib/kubelet/pods/906fd34c-5f61-43be-b273-22bae82fcbb8/volumes"
Jul 17 20:40:11.389122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:11.389083    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:40:11.847896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:11.847871    2572 scope.go:117] "RemoveContainer" containerID="01d69e2ce823ebc3e5451f6f4937759f1f62ed948e010a1723efaf8c3ecddf68"
Jul 17 20:40:12.407313 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:12.407224    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" event={"ID":"ef07d87a-6ba8-4142-88b2-79b9b37a99f3","Type":"ContainerStarted","Data":"58c4314c157b3e1e95d6a6f8b98e4501d3ad49839b99f8eee8191fecac3ed87f"}
Jul 17 20:40:13.413564 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:13.413527    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerID="58c4314c157b3e1e95d6a6f8b98e4501d3ad49839b99f8eee8191fecac3ed87f" exitCode=6
Jul 17 20:40:13.413951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:13.413601    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" event={"ID":"ef07d87a-6ba8-4142-88b2-79b9b37a99f3","Type":"ContainerDied","Data":"58c4314c157b3e1e95d6a6f8b98e4501d3ad49839b99f8eee8191fecac3ed87f"}
Jul 17 20:40:13.413951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:13.413688    2572 scope.go:117] "RemoveContainer" containerID="01d69e2ce823ebc3e5451f6f4937759f1f62ed948e010a1723efaf8c3ecddf68"
Jul 17 20:40:13.413951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:13.413935    2572 scope.go:117] "RemoveContainer" containerID="58c4314c157b3e1e95d6a6f8b98e4501d3ad49839b99f8eee8191fecac3ed87f"
Jul 17 20:40:13.414183 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:13.414156    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw_odh-ai-gateway-infra(ef07d87a-6ba8-4142-88b2-79b9b37a99f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3"
Jul 17 20:40:13.434604 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:13.434583    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"]
Jul 17 20:40:14.556265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:14.556242    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"
Jul 17 20:40:14.669285 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:14.669253    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07d87a-6ba8-4142-88b2-79b9b37a99f3-maas-api-service-ca\") pod \"ef07d87a-6ba8-4142-88b2-79b9b37a99f3\" (UID: \"ef07d87a-6ba8-4142-88b2-79b9b37a99f3\") "
Jul 17 20:40:14.669599 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:14.669578    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef07d87a-6ba8-4142-88b2-79b9b37a99f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef07d87a-6ba8-4142-88b2-79b9b37a99f3" (UID: "ef07d87a-6ba8-4142-88b2-79b9b37a99f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:40:14.770729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:14.770690    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07d87a-6ba8-4142-88b2-79b9b37a99f3-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:40:15.277788 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.277748    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"]
Jul 17 20:40:15.278155 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278139    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278159    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278175    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278185    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278204    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278214    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278224    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278233    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278246    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278256    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278333    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:15.278701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278346    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278357    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:15.278701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.278368    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:15.281251 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.281231    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"
Jul 17 20:40:15.288661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.288636    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"]
Jul 17 20:40:15.376525 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.376486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89f9d68a-c708-4d38-99c2-4624d8f68d3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-smlrx\" (UID: \"89f9d68a-c708-4d38-99c2-4624d8f68d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"
Jul 17 20:40:15.426427 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.426396    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"
Jul 17 20:40:15.426620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.426427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw" event={"ID":"ef07d87a-6ba8-4142-88b2-79b9b37a99f3","Type":"ContainerDied","Data":"2c85b2242da8604b9ef9ce8c688428a6a840244464d760335eadee18e25cf9ce"}
Jul 17 20:40:15.426620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.426494    2572 scope.go:117] "RemoveContainer" containerID="58c4314c157b3e1e95d6a6f8b98e4501d3ad49839b99f8eee8191fecac3ed87f"
Jul 17 20:40:15.452095 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.452068    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"]
Jul 17 20:40:15.458167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.458142    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kpzbw"]
Jul 17 20:40:15.477685 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.477658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89f9d68a-c708-4d38-99c2-4624d8f68d3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-smlrx\" (UID: \"89f9d68a-c708-4d38-99c2-4624d8f68d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"
Jul 17 20:40:15.478321 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.478304    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89f9d68a-c708-4d38-99c2-4624d8f68d3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-smlrx\" (UID: \"89f9d68a-c708-4d38-99c2-4624d8f68d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"
Jul 17 20:40:15.591910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.591818    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"
Jul 17 20:40:15.725444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.725340    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"]
Jul 17 20:40:15.728040 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:40:15.728010    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f9d68a_c708_4d38_99c2_4624d8f68d3a.slice/crio-6f6b9b8292ceeaf5af1cb0936db33f314db96d5d938050b45399d3c65dd4fb78 WatchSource:0}: Error finding container 6f6b9b8292ceeaf5af1cb0936db33f314db96d5d938050b45399d3c65dd4fb78: Status 404 returned error can't find the container with id 6f6b9b8292ceeaf5af1cb0936db33f314db96d5d938050b45399d3c65dd4fb78
Jul 17 20:40:15.850289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.850225    2572 scope.go:117] "RemoveContainer" containerID="bf0b86adec87b5653809ff19716bdca9c11ce6a47031dd9cd02c50d2476b1f9e"
Jul 17 20:40:15.853655 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:15.853635    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" path="/var/lib/kubelet/pods/ef07d87a-6ba8-4142-88b2-79b9b37a99f3/volumes"
Jul 17 20:40:16.433876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:16.433845    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" event={"ID":"78b94e51-ea1f-49e9-bd1f-7e18dbce535f","Type":"ContainerStarted","Data":"46d600b4449fe511be8c31f05f0a2817273d5d3a5bc1deb22265ea56eb7f23f8"}
Jul 17 20:40:16.435461 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:16.435430    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" event={"ID":"89f9d68a-c708-4d38-99c2-4624d8f68d3a","Type":"ContainerStarted","Data":"bdcc2ceb1c2f42cea11aa123744371f647766079444eabc5977a00c9af403060"}
Jul 17 20:40:16.435577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:16.435492    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" event={"ID":"89f9d68a-c708-4d38-99c2-4624d8f68d3a","Type":"ContainerStarted","Data":"6f6b9b8292ceeaf5af1cb0936db33f314db96d5d938050b45399d3c65dd4fb78"}
Jul 17 20:40:16.466898 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:16.466852    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" podStartSLOduration=1.466838299 podStartE2EDuration="1.466838299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:40:16.464611073 +0000 UTC m=+2651.158587223" watchObservedRunningTime="2026-07-17 20:40:16.466838299 +0000 UTC m=+2651.160814461"
Jul 17 20:40:17.442812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.442774    2572 generic.go:356] "Generic (PLEG): container finished" podID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerID="46d600b4449fe511be8c31f05f0a2817273d5d3a5bc1deb22265ea56eb7f23f8" exitCode=6
Jul 17 20:40:17.443277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.442844    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" event={"ID":"78b94e51-ea1f-49e9-bd1f-7e18dbce535f","Type":"ContainerDied","Data":"46d600b4449fe511be8c31f05f0a2817273d5d3a5bc1deb22265ea56eb7f23f8"}
Jul 17 20:40:17.443277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.442927    2572 scope.go:117] "RemoveContainer" containerID="bf0b86adec87b5653809ff19716bdca9c11ce6a47031dd9cd02c50d2476b1f9e"
Jul 17 20:40:17.443277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.443221    2572 scope.go:117] "RemoveContainer" containerID="46d600b4449fe511be8c31f05f0a2817273d5d3a5bc1deb22265ea56eb7f23f8"
Jul 17 20:40:17.443505 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:17.443466    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq_odh-ai-gateway-infra(78b94e51-ea1f-49e9-bd1f-7e18dbce535f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f"
Jul 17 20:40:17.444598 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.444575    2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerID="bdcc2ceb1c2f42cea11aa123744371f647766079444eabc5977a00c9af403060" exitCode=6
Jul 17 20:40:17.444735 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.444658    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" event={"ID":"89f9d68a-c708-4d38-99c2-4624d8f68d3a","Type":"ContainerDied","Data":"bdcc2ceb1c2f42cea11aa123744371f647766079444eabc5977a00c9af403060"}
Jul 17 20:40:17.444809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.444768    2572 scope.go:117] "RemoveContainer" containerID="bdcc2ceb1c2f42cea11aa123744371f647766079444eabc5977a00c9af403060"
Jul 17 20:40:17.461447 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:17.461423    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"]
Jul 17 20:40:18.453120 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.453091    2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerID="dbe9f5feb863dc6a35e472f954ba6ce344e204c78865a92d1e58d1fcc6974664" exitCode=6
Jul 17 20:40:18.453482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.453160    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" event={"ID":"89f9d68a-c708-4d38-99c2-4624d8f68d3a","Type":"ContainerDied","Data":"dbe9f5feb863dc6a35e472f954ba6ce344e204c78865a92d1e58d1fcc6974664"}
Jul 17 20:40:18.453482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.453233    2572 scope.go:117] "RemoveContainer" containerID="bdcc2ceb1c2f42cea11aa123744371f647766079444eabc5977a00c9af403060"
Jul 17 20:40:18.453482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.453369    2572 scope.go:117] "RemoveContainer" containerID="dbe9f5feb863dc6a35e472f954ba6ce344e204c78865a92d1e58d1fcc6974664"
Jul 17 20:40:18.453617 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:18.453582    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-smlrx_odh-ai-gateway-infra(89f9d68a-c708-4d38-99c2-4624d8f68d3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a"
Jul 17 20:40:18.584699 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.584675    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"
Jul 17 20:40:18.707496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.707399    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b94e51-ea1f-49e9-bd1f-7e18dbce535f-maas-api-service-ca\") pod \"78b94e51-ea1f-49e9-bd1f-7e18dbce535f\" (UID: \"78b94e51-ea1f-49e9-bd1f-7e18dbce535f\") "
Jul 17 20:40:18.707801 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.707777    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b94e51-ea1f-49e9-bd1f-7e18dbce535f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78b94e51-ea1f-49e9-bd1f-7e18dbce535f" (UID: "78b94e51-ea1f-49e9-bd1f-7e18dbce535f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:40:18.808842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:18.808805    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b94e51-ea1f-49e9-bd1f-7e18dbce535f-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:40:19.458482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:19.458453    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"
Jul 17 20:40:19.458482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:19.458471    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq" event={"ID":"78b94e51-ea1f-49e9-bd1f-7e18dbce535f","Type":"ContainerDied","Data":"eaf40dbe00d6e152d6f32b666666c40ae8991c514665967a61d947c0a1ba23d1"}
Jul 17 20:40:19.458981 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:19.458542    2572 scope.go:117] "RemoveContainer" containerID="46d600b4449fe511be8c31f05f0a2817273d5d3a5bc1deb22265ea56eb7f23f8"
Jul 17 20:40:19.460534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:19.460515    2572 scope.go:117] "RemoveContainer" containerID="dbe9f5feb863dc6a35e472f954ba6ce344e204c78865a92d1e58d1fcc6974664"
Jul 17 20:40:19.460722 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:19.460703    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-smlrx_odh-ai-gateway-infra(89f9d68a-c708-4d38-99c2-4624d8f68d3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a"
Jul 17 20:40:19.493194 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:19.493162    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"]
Jul 17 20:40:19.498926 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:19.498898    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-nsjwq"]
Jul 17 20:40:19.854263 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:19.854231    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" path="/var/lib/kubelet/pods/78b94e51-ea1f-49e9-bd1f-7e18dbce535f/volumes"
Jul 17 20:40:25.783187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:25.783144    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:40:29.073572 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.073526    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"]
Jul 17 20:40:29.074070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.073970    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:29.074070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.073989    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:29.074070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074001    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:29.074070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074009    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:29.074070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074039    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:29.074070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074065    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:29.074396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074165    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="906fd34c-5f61-43be-b273-22bae82fcbb8" containerName="revoke-keys"
Jul 17 20:40:29.074396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074178    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef07d87a-6ba8-4142-88b2-79b9b37a99f3" containerName="revoke-keys"
Jul 17 20:40:29.074396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074188    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:29.074396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074199    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:29.074396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.074207    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:29.078449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.078428    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"
Jul 17 20:40:29.084599 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.084570    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"]
Jul 17 20:40:29.204216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.204174    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d685b191-086f-4bd9-8ea9-800e8e073a17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr\" (UID: \"d685b191-086f-4bd9-8ea9-800e8e073a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"
Jul 17 20:40:29.304770 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.304727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d685b191-086f-4bd9-8ea9-800e8e073a17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr\" (UID: \"d685b191-086f-4bd9-8ea9-800e8e073a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"
Jul 17 20:40:29.305403 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.305383    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d685b191-086f-4bd9-8ea9-800e8e073a17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr\" (UID: \"d685b191-086f-4bd9-8ea9-800e8e073a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"
Jul 17 20:40:29.390697 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.390610    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"
Jul 17 20:40:29.520105 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:29.520069    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"]
Jul 17 20:40:29.522909 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:40:29.522874    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd685b191_086f_4bd9_8ea9_800e8e073a17.slice/crio-f4f1dc6d250355b3c6ee3b3b4ed0a014414e33f4d5b2d76f95866a6b34afd18c WatchSource:0}: Error finding container f4f1dc6d250355b3c6ee3b3b4ed0a014414e33f4d5b2d76f95866a6b34afd18c: Status 404 returned error can't find the container with id f4f1dc6d250355b3c6ee3b3b4ed0a014414e33f4d5b2d76f95866a6b34afd18c
Jul 17 20:40:30.513064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:30.513022    2572 generic.go:356] "Generic (PLEG): container finished" podID="d685b191-086f-4bd9-8ea9-800e8e073a17" containerID="1eb7cb844c5b0032fa5195a101e8e1e90eef4db22f60275fa4a63dbec849c074" exitCode=0
Jul 17 20:40:30.513470 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:30.513089    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr" event={"ID":"d685b191-086f-4bd9-8ea9-800e8e073a17","Type":"ContainerDied","Data":"1eb7cb844c5b0032fa5195a101e8e1e90eef4db22f60275fa4a63dbec849c074"}
Jul 17 20:40:30.513470 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:30.513146    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr" event={"ID":"d685b191-086f-4bd9-8ea9-800e8e073a17","Type":"ContainerStarted","Data":"f4f1dc6d250355b3c6ee3b3b4ed0a014414e33f4d5b2d76f95866a6b34afd18c"}
Jul 17 20:40:31.657204 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:31.657181    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"
Jul 17 20:40:31.725777 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:31.725749    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d685b191-086f-4bd9-8ea9-800e8e073a17-maas-api-service-ca\") pod \"d685b191-086f-4bd9-8ea9-800e8e073a17\" (UID: \"d685b191-086f-4bd9-8ea9-800e8e073a17\") "
Jul 17 20:40:31.726142 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:31.726122    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d685b191-086f-4bd9-8ea9-800e8e073a17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d685b191-086f-4bd9-8ea9-800e8e073a17" (UID: "d685b191-086f-4bd9-8ea9-800e8e073a17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:40:31.827296 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:31.827214    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d685b191-086f-4bd9-8ea9-800e8e073a17-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:40:32.523485 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:32.523448    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr" event={"ID":"d685b191-086f-4bd9-8ea9-800e8e073a17","Type":"ContainerDied","Data":"f4f1dc6d250355b3c6ee3b3b4ed0a014414e33f4d5b2d76f95866a6b34afd18c"}
Jul 17 20:40:32.523485 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:32.523485    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr"
Jul 17 20:40:32.523722 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:32.523506    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4f1dc6d250355b3c6ee3b3b4ed0a014414e33f4d5b2d76f95866a6b34afd18c"
Jul 17 20:40:33.848424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:33.848390    2572 scope.go:117] "RemoveContainer" containerID="dbe9f5feb863dc6a35e472f954ba6ce344e204c78865a92d1e58d1fcc6974664"
Jul 17 20:40:34.537110 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:34.537078    2572 generic.go:356] "Generic (PLEG): container finished" podID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerID="b39b94dc1ee426fea2dbc51e39d3adc201a8b8cbd45d17d89d34c177091b8e85" exitCode=6
Jul 17 20:40:34.537308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:34.537133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" event={"ID":"89f9d68a-c708-4d38-99c2-4624d8f68d3a","Type":"ContainerDied","Data":"b39b94dc1ee426fea2dbc51e39d3adc201a8b8cbd45d17d89d34c177091b8e85"}
Jul 17 20:40:34.537308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:34.537202    2572 scope.go:117] "RemoveContainer" containerID="dbe9f5feb863dc6a35e472f954ba6ce344e204c78865a92d1e58d1fcc6974664"
Jul 17 20:40:34.537604 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:34.537584    2572 scope.go:117] "RemoveContainer" containerID="b39b94dc1ee426fea2dbc51e39d3adc201a8b8cbd45d17d89d34c177091b8e85"
Jul 17 20:40:34.537805 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:34.537786    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-smlrx_odh-ai-gateway-infra(89f9d68a-c708-4d38-99c2-4624d8f68d3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a"
Jul 17 20:40:35.562570 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:35.562537    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"]
Jul 17 20:40:35.696093 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:35.696073    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"
Jul 17 20:40:35.758081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:35.758029    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89f9d68a-c708-4d38-99c2-4624d8f68d3a-maas-api-service-ca\") pod \"89f9d68a-c708-4d38-99c2-4624d8f68d3a\" (UID: \"89f9d68a-c708-4d38-99c2-4624d8f68d3a\") "
Jul 17 20:40:35.758384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:35.758360    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f9d68a-c708-4d38-99c2-4624d8f68d3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89f9d68a-c708-4d38-99c2-4624d8f68d3a" (UID: "89f9d68a-c708-4d38-99c2-4624d8f68d3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:40:35.858693 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:35.858618    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89f9d68a-c708-4d38-99c2-4624d8f68d3a-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:40:36.549068 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:36.549016    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"
Jul 17 20:40:36.549068 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:36.549013    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx" event={"ID":"89f9d68a-c708-4d38-99c2-4624d8f68d3a","Type":"ContainerDied","Data":"6f6b9b8292ceeaf5af1cb0936db33f314db96d5d938050b45399d3c65dd4fb78"}
Jul 17 20:40:36.549314 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:36.549111    2572 scope.go:117] "RemoveContainer" containerID="b39b94dc1ee426fea2dbc51e39d3adc201a8b8cbd45d17d89d34c177091b8e85"
Jul 17 20:40:36.564618 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:36.564592    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"]
Jul 17 20:40:36.572380 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:36.572357    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-smlrx"]
Jul 17 20:40:37.853362 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:37.853328    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" path="/var/lib/kubelet/pods/89f9d68a-c708-4d38-99c2-4624d8f68d3a/volumes"
Jul 17 20:40:41.291220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291185    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"]
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291479    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291490    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b94e51-ea1f-49e9-bd1f-7e18dbce535f" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291500    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291505    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291513    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d685b191-086f-4bd9-8ea9-800e8e073a17" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291519    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d685b191-086f-4bd9-8ea9-800e8e073a17" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291530    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291535    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291541    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291546    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291591    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291601    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:41.291609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.291608    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d685b191-086f-4bd9-8ea9-800e8e073a17" containerName="revoke-keys"
Jul 17 20:40:41.295868 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.295851    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"
Jul 17 20:40:41.298268 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.298249    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:40:41.298322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.298250    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:40:41.302143 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.302119    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"]
Jul 17 20:40:41.406031 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.405991    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-rblgr\" (UID: \"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"
Jul 17 20:40:41.506623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.506584    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-rblgr\" (UID: \"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"
Jul 17 20:40:41.507305 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.507285    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-rblgr\" (UID: \"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"
Jul 17 20:40:41.607641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.607564    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"
Jul 17 20:40:41.734675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:41.734633    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"]
Jul 17 20:40:41.737473 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:40:41.737443    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94eaaa1d_a82d_4b14_bcf7_de3d0d2e9263.slice/crio-6b616577820786ab9be2c9b42eef766346e802415e80725bd1fafa0cfcbfc0e7 WatchSource:0}: Error finding container 6b616577820786ab9be2c9b42eef766346e802415e80725bd1fafa0cfcbfc0e7: Status 404 returned error can't find the container with id 6b616577820786ab9be2c9b42eef766346e802415e80725bd1fafa0cfcbfc0e7
Jul 17 20:40:42.577019 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:42.576983    2572 generic.go:356] "Generic (PLEG): container finished" podID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerID="6c34a3117eddcc30ec6c77189149b5f3c2c732dcc0bcfca44ba5dbfad7bcb4cd" exitCode=6
Jul 17 20:40:42.577445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:42.577034    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" event={"ID":"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263","Type":"ContainerDied","Data":"6c34a3117eddcc30ec6c77189149b5f3c2c732dcc0bcfca44ba5dbfad7bcb4cd"}
Jul 17 20:40:42.577445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:42.577100    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" event={"ID":"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263","Type":"ContainerStarted","Data":"6b616577820786ab9be2c9b42eef766346e802415e80725bd1fafa0cfcbfc0e7"}
Jul 17 20:40:42.577445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:42.577256    2572 scope.go:117] "RemoveContainer" containerID="6c34a3117eddcc30ec6c77189149b5f3c2c732dcc0bcfca44ba5dbfad7bcb4cd"
Jul 17 20:40:43.377018 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:43.376981    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:40:43.583723 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:43.583688    2572 generic.go:356] "Generic (PLEG): container finished" podID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerID="823061813b56a5248ae5735b5d45ae162169d1179964cd63381b3375479ff3d4" exitCode=6
Jul 17 20:40:43.584206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:43.583730    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" event={"ID":"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263","Type":"ContainerDied","Data":"823061813b56a5248ae5735b5d45ae162169d1179964cd63381b3375479ff3d4"}
Jul 17 20:40:43.584206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:43.583776    2572 scope.go:117] "RemoveContainer" containerID="6c34a3117eddcc30ec6c77189149b5f3c2c732dcc0bcfca44ba5dbfad7bcb4cd"
Jul 17 20:40:43.584206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:43.583950    2572 scope.go:117] "RemoveContainer" containerID="823061813b56a5248ae5735b5d45ae162169d1179964cd63381b3375479ff3d4"
Jul 17 20:40:43.584361 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:43.584201    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-rblgr_odh-ai-gateway-infra(94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263"
Jul 17 20:40:44.589184 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:44.589155    2572 scope.go:117] "RemoveContainer" containerID="823061813b56a5248ae5735b5d45ae162169d1179964cd63381b3375479ff3d4"
Jul 17 20:40:44.589572 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:44.589334    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-rblgr_odh-ai-gateway-infra(94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263"
Jul 17 20:40:47.502471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.502438    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"]
Jul 17 20:40:47.502862 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.502804    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f9d68a-c708-4d38-99c2-4624d8f68d3a" containerName="revoke-keys"
Jul 17 20:40:47.505303 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.505287    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"
Jul 17 20:40:47.514009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.513980    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"]
Jul 17 20:40:47.659011 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.658952    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872146c4-de15-46e6-93bb-0024ba5748a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx\" (UID: \"872146c4-de15-46e6-93bb-0024ba5748a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"
Jul 17 20:40:47.759610 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.759500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872146c4-de15-46e6-93bb-0024ba5748a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx\" (UID: \"872146c4-de15-46e6-93bb-0024ba5748a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"
Jul 17 20:40:47.760160 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.760143    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872146c4-de15-46e6-93bb-0024ba5748a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx\" (UID: \"872146c4-de15-46e6-93bb-0024ba5748a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"
Jul 17 20:40:47.816648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.816617    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"
Jul 17 20:40:47.945536 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:47.945510    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"]
Jul 17 20:40:47.948138 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:40:47.948101    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872146c4_de15_46e6_93bb_0024ba5748a2.slice/crio-1e9ab9feed0906a9df27ce65e999fa5109c9eba6f47bbcd57b195205979c8871 WatchSource:0}: Error finding container 1e9ab9feed0906a9df27ce65e999fa5109c9eba6f47bbcd57b195205979c8871: Status 404 returned error can't find the container with id 1e9ab9feed0906a9df27ce65e999fa5109c9eba6f47bbcd57b195205979c8871
Jul 17 20:40:48.608083 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:48.608035    2572 generic.go:356] "Generic (PLEG): container finished" podID="872146c4-de15-46e6-93bb-0024ba5748a2" containerID="4ebc5876c44ba4d3ec3288467aa18230dc5ea86bf55a12b40bf4085951224915" exitCode=6
Jul 17 20:40:48.608449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:48.608136    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" event={"ID":"872146c4-de15-46e6-93bb-0024ba5748a2","Type":"ContainerDied","Data":"4ebc5876c44ba4d3ec3288467aa18230dc5ea86bf55a12b40bf4085951224915"}
Jul 17 20:40:48.608449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:48.608189    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" event={"ID":"872146c4-de15-46e6-93bb-0024ba5748a2","Type":"ContainerStarted","Data":"1e9ab9feed0906a9df27ce65e999fa5109c9eba6f47bbcd57b195205979c8871"}
Jul 17 20:40:48.608449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:48.608317    2572 scope.go:117] "RemoveContainer" containerID="4ebc5876c44ba4d3ec3288467aa18230dc5ea86bf55a12b40bf4085951224915"
Jul 17 20:40:49.614834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:49.614749    2572 generic.go:356] "Generic (PLEG): container finished" podID="872146c4-de15-46e6-93bb-0024ba5748a2" containerID="cdd84899ab30455a5d0ca442a357a87dcafbe738f12741dfd74e6f778a315ec1" exitCode=6
Jul 17 20:40:49.615237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:49.614833    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" event={"ID":"872146c4-de15-46e6-93bb-0024ba5748a2","Type":"ContainerDied","Data":"cdd84899ab30455a5d0ca442a357a87dcafbe738f12741dfd74e6f778a315ec1"}
Jul 17 20:40:49.615237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:49.614911    2572 scope.go:117] "RemoveContainer" containerID="4ebc5876c44ba4d3ec3288467aa18230dc5ea86bf55a12b40bf4085951224915"
Jul 17 20:40:49.615237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:49.615110    2572 scope.go:117] "RemoveContainer" containerID="cdd84899ab30455a5d0ca442a357a87dcafbe738f12741dfd74e6f778a315ec1"
Jul 17 20:40:49.615388 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:49.615317    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx_odh-ai-gateway-infra(872146c4-de15-46e6-93bb-0024ba5748a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" podUID="872146c4-de15-46e6-93bb-0024ba5748a2"
Jul 17 20:40:50.621023 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:50.620998    2572 scope.go:117] "RemoveContainer" containerID="cdd84899ab30455a5d0ca442a357a87dcafbe738f12741dfd74e6f778a315ec1"
Jul 17 20:40:50.621416 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:40:50.621198    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx_odh-ai-gateway-infra(872146c4-de15-46e6-93bb-0024ba5748a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" podUID="872146c4-de15-46e6-93bb-0024ba5748a2"
Jul 17 20:40:59.383523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.383439    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"]
Jul 17 20:40:59.387911 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.387892    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"
Jul 17 20:40:59.393569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.393540    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"]
Jul 17 20:40:59.559180 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.559143    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18141782-2467-43f4-95e5-90f640e389b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk\" (UID: \"18141782-2467-43f4-95e5-90f640e389b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"
Jul 17 20:40:59.660282 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.660206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18141782-2467-43f4-95e5-90f640e389b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk\" (UID: \"18141782-2467-43f4-95e5-90f640e389b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"
Jul 17 20:40:59.660865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.660844    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18141782-2467-43f4-95e5-90f640e389b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk\" (UID: \"18141782-2467-43f4-95e5-90f640e389b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"
Jul 17 20:40:59.699799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.699767    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"
Jul 17 20:40:59.828304 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.828280    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"]
Jul 17 20:40:59.830963 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:40:59.830935    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18141782_2467_43f4_95e5_90f640e389b9.slice/crio-f385a98a2b306da27e97d2704b37b08ab62c28581c76aca4b0fd1057c27421c4 WatchSource:0}: Error finding container f385a98a2b306da27e97d2704b37b08ab62c28581c76aca4b0fd1057c27421c4: Status 404 returned error can't find the container with id f385a98a2b306da27e97d2704b37b08ab62c28581c76aca4b0fd1057c27421c4
Jul 17 20:40:59.848238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:40:59.848220    2572 scope.go:117] "RemoveContainer" containerID="823061813b56a5248ae5735b5d45ae162169d1179964cd63381b3375479ff3d4"
Jul 17 20:41:00.667360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.667323    2572 generic.go:356] "Generic (PLEG): container finished" podID="18141782-2467-43f4-95e5-90f640e389b9" containerID="b4ecc5f6a55871515cbae2749cb64d2251ab66d025cd6231112b0eb366c5fe54" exitCode=6
Jul 17 20:41:00.667821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.667380    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" event={"ID":"18141782-2467-43f4-95e5-90f640e389b9","Type":"ContainerDied","Data":"b4ecc5f6a55871515cbae2749cb64d2251ab66d025cd6231112b0eb366c5fe54"}
Jul 17 20:41:00.667821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.667435    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" event={"ID":"18141782-2467-43f4-95e5-90f640e389b9","Type":"ContainerStarted","Data":"f385a98a2b306da27e97d2704b37b08ab62c28581c76aca4b0fd1057c27421c4"}
Jul 17 20:41:00.667821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.667648    2572 scope.go:117] "RemoveContainer" containerID="b4ecc5f6a55871515cbae2749cb64d2251ab66d025cd6231112b0eb366c5fe54"
Jul 17 20:41:00.669339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.669318    2572 generic.go:356] "Generic (PLEG): container finished" podID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerID="5d8952680992444a855a88092f65cbe984d4c7d369e5c1a2eec5f5ccd107144f" exitCode=6
Jul 17 20:41:00.669468 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.669345    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" event={"ID":"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263","Type":"ContainerDied","Data":"5d8952680992444a855a88092f65cbe984d4c7d369e5c1a2eec5f5ccd107144f"}
Jul 17 20:41:00.669468 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.669399    2572 scope.go:117] "RemoveContainer" containerID="823061813b56a5248ae5735b5d45ae162169d1179964cd63381b3375479ff3d4"
Jul 17 20:41:00.669703 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:00.669675    2572 scope.go:117] "RemoveContainer" containerID="5d8952680992444a855a88092f65cbe984d4c7d369e5c1a2eec5f5ccd107144f"
Jul 17 20:41:00.669865 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:00.669844    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-rblgr_odh-ai-gateway-infra(94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263"
Jul 17 20:41:01.675861 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.675758    2572 generic.go:356] "Generic (PLEG): container finished" podID="18141782-2467-43f4-95e5-90f640e389b9" containerID="9a0c45bf662f402cbf7711bc141e72144f13ae0ddfbd649307186a05a6a505a4" exitCode=6
Jul 17 20:41:01.675861 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.675843    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" event={"ID":"18141782-2467-43f4-95e5-90f640e389b9","Type":"ContainerDied","Data":"9a0c45bf662f402cbf7711bc141e72144f13ae0ddfbd649307186a05a6a505a4"}
Jul 17 20:41:01.676355 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.675900    2572 scope.go:117] "RemoveContainer" containerID="b4ecc5f6a55871515cbae2749cb64d2251ab66d025cd6231112b0eb366c5fe54"
Jul 17 20:41:01.676355 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.676097    2572 scope.go:117] "RemoveContainer" containerID="9a0c45bf662f402cbf7711bc141e72144f13ae0ddfbd649307186a05a6a505a4"
Jul 17 20:41:01.676355 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:01.676279    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk_odh-ai-gateway-infra(18141782-2467-43f4-95e5-90f640e389b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" podUID="18141782-2467-43f4-95e5-90f640e389b9"
Jul 17 20:41:01.710409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.710376    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"]
Jul 17 20:41:01.848625 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.848603    2572 scope.go:117] "RemoveContainer" containerID="cdd84899ab30455a5d0ca442a357a87dcafbe738f12741dfd74e6f778a315ec1"
Jul 17 20:41:01.849146 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.849121    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"
Jul 17 20:41:01.979022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.978983    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263-maas-api-service-ca\") pod \"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263\" (UID: \"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263\") "
Jul 17 20:41:01.979496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:01.979397    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" (UID: "94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:41:02.080479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.080440    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:41:02.683855 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.683756    2572 generic.go:356] "Generic (PLEG): container finished" podID="872146c4-de15-46e6-93bb-0024ba5748a2" containerID="7fd61c273747366631befd691958d6dae76b1508c88f3ecf625c4872e6eaee87" exitCode=6
Jul 17 20:41:02.683855 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.683829    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" event={"ID":"872146c4-de15-46e6-93bb-0024ba5748a2","Type":"ContainerDied","Data":"7fd61c273747366631befd691958d6dae76b1508c88f3ecf625c4872e6eaee87"}
Jul 17 20:41:02.684398 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.683882    2572 scope.go:117] "RemoveContainer" containerID="cdd84899ab30455a5d0ca442a357a87dcafbe738f12741dfd74e6f778a315ec1"
Jul 17 20:41:02.684398 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.684235    2572 scope.go:117] "RemoveContainer" containerID="7fd61c273747366631befd691958d6dae76b1508c88f3ecf625c4872e6eaee87"
Jul 17 20:41:02.684483 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:02.684422    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx_odh-ai-gateway-infra(872146c4-de15-46e6-93bb-0024ba5748a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" podUID="872146c4-de15-46e6-93bb-0024ba5748a2"
Jul 17 20:41:02.685654 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.685635    2572 scope.go:117] "RemoveContainer" containerID="9a0c45bf662f402cbf7711bc141e72144f13ae0ddfbd649307186a05a6a505a4"
Jul 17 20:41:02.685806 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:02.685786    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk_odh-ai-gateway-infra(18141782-2467-43f4-95e5-90f640e389b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" podUID="18141782-2467-43f4-95e5-90f640e389b9"
Jul 17 20:41:02.686629 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.686607    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr" event={"ID":"94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263","Type":"ContainerDied","Data":"6b616577820786ab9be2c9b42eef766346e802415e80725bd1fafa0cfcbfc0e7"}
Jul 17 20:41:02.686705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.686628    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"
Jul 17 20:41:02.694497 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.694483    2572 scope.go:117] "RemoveContainer" containerID="5d8952680992444a855a88092f65cbe984d4c7d369e5c1a2eec5f5ccd107144f"
Jul 17 20:41:02.710384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.710361    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"]
Jul 17 20:41:02.714215 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:02.714192    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rblgr"]
Jul 17 20:41:03.708658 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:03.708624    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"]
Jul 17 20:41:03.845055 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:03.845029    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"
Jul 17 20:41:03.853434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:03.853412    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" path="/var/lib/kubelet/pods/94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263/volumes"
Jul 17 20:41:03.995413 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:03.995383    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872146c4-de15-46e6-93bb-0024ba5748a2-maas-api-service-ca\") pod \"872146c4-de15-46e6-93bb-0024ba5748a2\" (UID: \"872146c4-de15-46e6-93bb-0024ba5748a2\") "
Jul 17 20:41:03.995739 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:03.995716    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872146c4-de15-46e6-93bb-0024ba5748a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "872146c4-de15-46e6-93bb-0024ba5748a2" (UID: "872146c4-de15-46e6-93bb-0024ba5748a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:41:04.096709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:04.096671    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/872146c4-de15-46e6-93bb-0024ba5748a2-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:41:04.698555 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:04.698508    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx" event={"ID":"872146c4-de15-46e6-93bb-0024ba5748a2","Type":"ContainerDied","Data":"1e9ab9feed0906a9df27ce65e999fa5109c9eba6f47bbcd57b195205979c8871"}
Jul 17 20:41:04.698754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:04.698570    2572 scope.go:117] "RemoveContainer" containerID="7fd61c273747366631befd691958d6dae76b1508c88f3ecf625c4872e6eaee87"
Jul 17 20:41:04.698754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:04.698524    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"
Jul 17 20:41:04.721261 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:04.721238    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"]
Jul 17 20:41:04.726442 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:04.726418    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-kzjfx"]
Jul 17 20:41:05.853895 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:05.853863    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" path="/var/lib/kubelet/pods/872146c4-de15-46e6-93bb-0024ba5748a2/volumes"
Jul 17 20:41:15.322822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.322782    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"]
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323125    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323137    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323151    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323156    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323170    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323176    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323188    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:15.323193 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323196    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:15.323501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323206    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323211    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323309    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323321    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:15.323501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323336    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:15.323501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.323341    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:15.326201 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.326183    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"
Jul 17 20:41:15.333147 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.332811    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"]
Jul 17 20:41:15.371518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.371478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab9170-b00d-45db-919f-eea72c5254f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg\" (UID: \"a7ab9170-b00d-45db-919f-eea72c5254f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"
Jul 17 20:41:15.472115 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.472080    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab9170-b00d-45db-919f-eea72c5254f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg\" (UID: \"a7ab9170-b00d-45db-919f-eea72c5254f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"
Jul 17 20:41:15.472650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.472632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab9170-b00d-45db-919f-eea72c5254f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg\" (UID: \"a7ab9170-b00d-45db-919f-eea72c5254f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"
Jul 17 20:41:15.636965 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.636870    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"
Jul 17 20:41:15.764405 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.764256    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"]
Jul 17 20:41:15.767246 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:41:15.767218    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ab9170_b00d_45db_919f_eea72c5254f8.slice/crio-6a2246d18fb7b7f3ec28290c44ee53869cbb9e4f8241c5cb5d8976f5da5de052 WatchSource:0}: Error finding container 6a2246d18fb7b7f3ec28290c44ee53869cbb9e4f8241c5cb5d8976f5da5de052: Status 404 returned error can't find the container with id 6a2246d18fb7b7f3ec28290c44ee53869cbb9e4f8241c5cb5d8976f5da5de052
Jul 17 20:41:15.851490 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:15.851465    2572 scope.go:117] "RemoveContainer" containerID="9a0c45bf662f402cbf7711bc141e72144f13ae0ddfbd649307186a05a6a505a4"
Jul 17 20:41:16.753412 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.753378    2572 generic.go:356] "Generic (PLEG): container finished" podID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerID="d05b13347e15e58cedf53628e2884f4c212eea42e5bd4f6093e03861ce307590" exitCode=6
Jul 17 20:41:16.753829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.753428    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" event={"ID":"a7ab9170-b00d-45db-919f-eea72c5254f8","Type":"ContainerDied","Data":"d05b13347e15e58cedf53628e2884f4c212eea42e5bd4f6093e03861ce307590"}
Jul 17 20:41:16.753829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.753505    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" event={"ID":"a7ab9170-b00d-45db-919f-eea72c5254f8","Type":"ContainerStarted","Data":"6a2246d18fb7b7f3ec28290c44ee53869cbb9e4f8241c5cb5d8976f5da5de052"}
Jul 17 20:41:16.753829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.753685    2572 scope.go:117] "RemoveContainer" containerID="d05b13347e15e58cedf53628e2884f4c212eea42e5bd4f6093e03861ce307590"
Jul 17 20:41:16.755387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.755369    2572 generic.go:356] "Generic (PLEG): container finished" podID="18141782-2467-43f4-95e5-90f640e389b9" containerID="f6d5cb88763783a2d86bfaee5acb2b6e8e2b02dc28f2749c15f90492be089da3" exitCode=6
Jul 17 20:41:16.755507 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.755399    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" event={"ID":"18141782-2467-43f4-95e5-90f640e389b9","Type":"ContainerDied","Data":"f6d5cb88763783a2d86bfaee5acb2b6e8e2b02dc28f2749c15f90492be089da3"}
Jul 17 20:41:16.755507 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.755433    2572 scope.go:117] "RemoveContainer" containerID="9a0c45bf662f402cbf7711bc141e72144f13ae0ddfbd649307186a05a6a505a4"
Jul 17 20:41:16.755715 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:16.755693    2572 scope.go:117] "RemoveContainer" containerID="f6d5cb88763783a2d86bfaee5acb2b6e8e2b02dc28f2749c15f90492be089da3"
Jul 17 20:41:16.755954 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:16.755904    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk_odh-ai-gateway-infra(18141782-2467-43f4-95e5-90f640e389b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" podUID="18141782-2467-43f4-95e5-90f640e389b9"
Jul 17 20:41:17.762873 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.762842    2572 generic.go:356] "Generic (PLEG): container finished" podID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerID="aadb1bcf51c0a555b96e4a410aca7acc1700bf37f46bc39b51433213faf8d36e" exitCode=6
Jul 17 20:41:17.763383 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.762928    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" event={"ID":"a7ab9170-b00d-45db-919f-eea72c5254f8","Type":"ContainerDied","Data":"aadb1bcf51c0a555b96e4a410aca7acc1700bf37f46bc39b51433213faf8d36e"}
Jul 17 20:41:17.763383 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.762993    2572 scope.go:117] "RemoveContainer" containerID="d05b13347e15e58cedf53628e2884f4c212eea42e5bd4f6093e03861ce307590"
Jul 17 20:41:17.763383 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.763186    2572 scope.go:117] "RemoveContainer" containerID="aadb1bcf51c0a555b96e4a410aca7acc1700bf37f46bc39b51433213faf8d36e"
Jul 17 20:41:17.763501 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:17.763408    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg_odh-ai-gateway-infra(a7ab9170-b00d-45db-919f-eea72c5254f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8"
Jul 17 20:41:17.794758 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.794729    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"]
Jul 17 20:41:17.930004 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.929976    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"
Jul 17 20:41:17.991325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.991293    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18141782-2467-43f4-95e5-90f640e389b9-maas-api-service-ca\") pod \"18141782-2467-43f4-95e5-90f640e389b9\" (UID: \"18141782-2467-43f4-95e5-90f640e389b9\") "
Jul 17 20:41:17.991656 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:17.991631    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18141782-2467-43f4-95e5-90f640e389b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18141782-2467-43f4-95e5-90f640e389b9" (UID: "18141782-2467-43f4-95e5-90f640e389b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:41:18.091826 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:18.091734    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18141782-2467-43f4-95e5-90f640e389b9-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:41:18.768671 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:18.768639    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"
Jul 17 20:41:18.769165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:18.768639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk" event={"ID":"18141782-2467-43f4-95e5-90f640e389b9","Type":"ContainerDied","Data":"f385a98a2b306da27e97d2704b37b08ab62c28581c76aca4b0fd1057c27421c4"}
Jul 17 20:41:18.769165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:18.768782    2572 scope.go:117] "RemoveContainer" containerID="f6d5cb88763783a2d86bfaee5acb2b6e8e2b02dc28f2749c15f90492be089da3"
Jul 17 20:41:18.770617 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:18.770596    2572 scope.go:117] "RemoveContainer" containerID="aadb1bcf51c0a555b96e4a410aca7acc1700bf37f46bc39b51433213faf8d36e"
Jul 17 20:41:18.770835 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:18.770814    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg_odh-ai-gateway-infra(a7ab9170-b00d-45db-919f-eea72c5254f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8"
Jul 17 20:41:18.799990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:18.799964    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"]
Jul 17 20:41:18.806011 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:18.805987    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-64zkk"]
Jul 17 20:41:19.853408 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:19.853379    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18141782-2467-43f4-95e5-90f640e389b9" path="/var/lib/kubelet/pods/18141782-2467-43f4-95e5-90f640e389b9/volumes"
Jul 17 20:41:31.848731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:31.848701    2572 scope.go:117] "RemoveContainer" containerID="aadb1bcf51c0a555b96e4a410aca7acc1700bf37f46bc39b51433213faf8d36e"
Jul 17 20:41:32.835764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:32.835731    2572 generic.go:356] "Generic (PLEG): container finished" podID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerID="e54bf7e291a0bebe14e6b066f5d7f0e11aadd8ccb6cb9d659ee0b8d27885e734" exitCode=6
Jul 17 20:41:32.835988 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:32.835811    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" event={"ID":"a7ab9170-b00d-45db-919f-eea72c5254f8","Type":"ContainerDied","Data":"e54bf7e291a0bebe14e6b066f5d7f0e11aadd8ccb6cb9d659ee0b8d27885e734"}
Jul 17 20:41:32.835988 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:32.835886    2572 scope.go:117] "RemoveContainer" containerID="aadb1bcf51c0a555b96e4a410aca7acc1700bf37f46bc39b51433213faf8d36e"
Jul 17 20:41:32.836207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:32.836180    2572 scope.go:117] "RemoveContainer" containerID="e54bf7e291a0bebe14e6b066f5d7f0e11aadd8ccb6cb9d659ee0b8d27885e734"
Jul 17 20:41:32.836438 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:32.836415    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg_odh-ai-gateway-infra(a7ab9170-b00d-45db-919f-eea72c5254f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8"
Jul 17 20:41:33.864564 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:33.864528    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"]
Jul 17 20:41:34.009631 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.009607    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"
Jul 17 20:41:34.105070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.105018    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab9170-b00d-45db-919f-eea72c5254f8-maas-api-service-ca\") pod \"a7ab9170-b00d-45db-919f-eea72c5254f8\" (UID: \"a7ab9170-b00d-45db-919f-eea72c5254f8\") "
Jul 17 20:41:34.105409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.105384    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab9170-b00d-45db-919f-eea72c5254f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7ab9170-b00d-45db-919f-eea72c5254f8" (UID: "a7ab9170-b00d-45db-919f-eea72c5254f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:41:34.205661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.205576    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab9170-b00d-45db-919f-eea72c5254f8-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:41:34.846668 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.846627    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg" event={"ID":"a7ab9170-b00d-45db-919f-eea72c5254f8","Type":"ContainerDied","Data":"6a2246d18fb7b7f3ec28290c44ee53869cbb9e4f8241c5cb5d8976f5da5de052"}
Jul 17 20:41:34.846859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.846696    2572 scope.go:117] "RemoveContainer" containerID="e54bf7e291a0bebe14e6b066f5d7f0e11aadd8ccb6cb9d659ee0b8d27885e734"
Jul 17 20:41:34.846859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.846652    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"
Jul 17 20:41:34.865522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.865497    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"]
Jul 17 20:41:34.869171 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:34.869140    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-cgmpg"]
Jul 17 20:41:35.852802 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:35.852771    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" path="/var/lib/kubelet/pods/a7ab9170-b00d-45db-919f-eea72c5254f8/volumes"
Jul 17 20:41:37.570103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.570064    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"]
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.570974    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571009    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571041    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571074    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571102    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571111    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571148    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571156    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571176    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571184    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571382    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571393    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="872146c4-de15-46e6-93bb-0024ba5748a2" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571410    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571419    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571430    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94eaaa1d-a82d-4b14-bcf7-de3d0d2e9263" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571440    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:37.572880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.571457    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:37.578488 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.578465    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"
Jul 17 20:41:37.581389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.581361    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:41:37.581534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.581407    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:41:37.582010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.581990    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"]
Jul 17 20:41:37.630019 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.629981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9c0f01-f474-421a-b7c7-41aa93eecaed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr\" (UID: \"8b9c0f01-f474-421a-b7c7-41aa93eecaed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"
Jul 17 20:41:37.730388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.730351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9c0f01-f474-421a-b7c7-41aa93eecaed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr\" (UID: \"8b9c0f01-f474-421a-b7c7-41aa93eecaed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"
Jul 17 20:41:37.731016 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.730995    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9c0f01-f474-421a-b7c7-41aa93eecaed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr\" (UID: \"8b9c0f01-f474-421a-b7c7-41aa93eecaed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"
Jul 17 20:41:37.890831 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:37.890743    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"
Jul 17 20:41:38.035640 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:38.035617    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"]
Jul 17 20:41:38.038288 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:41:38.038258    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9c0f01_f474_421a_b7c7_41aa93eecaed.slice/crio-e3cd8507bfd0c63b59cdd347ccfc315b75d9b207a325c3e4bdb739e5ca5206a2 WatchSource:0}: Error finding container e3cd8507bfd0c63b59cdd347ccfc315b75d9b207a325c3e4bdb739e5ca5206a2: Status 404 returned error can't find the container with id e3cd8507bfd0c63b59cdd347ccfc315b75d9b207a325c3e4bdb739e5ca5206a2
Jul 17 20:41:38.865060 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:38.865012    2572 generic.go:356] "Generic (PLEG): container finished" podID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerID="7141a450ba4b4363f06f3baaae0c2a1ba5e1b9975d133d07654cde8e92fbfaeb" exitCode=6
Jul 17 20:41:38.865533 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:38.865101    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" event={"ID":"8b9c0f01-f474-421a-b7c7-41aa93eecaed","Type":"ContainerDied","Data":"7141a450ba4b4363f06f3baaae0c2a1ba5e1b9975d133d07654cde8e92fbfaeb"}
Jul 17 20:41:38.865533 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:38.865144    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" event={"ID":"8b9c0f01-f474-421a-b7c7-41aa93eecaed","Type":"ContainerStarted","Data":"e3cd8507bfd0c63b59cdd347ccfc315b75d9b207a325c3e4bdb739e5ca5206a2"}
Jul 17 20:41:38.865533 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:38.865313    2572 scope.go:117] "RemoveContainer" containerID="7141a450ba4b4363f06f3baaae0c2a1ba5e1b9975d133d07654cde8e92fbfaeb"
Jul 17 20:41:39.871177 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:39.871087    2572 generic.go:356] "Generic (PLEG): container finished" podID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerID="f500a06020daa831ab4f6df4eb497695c8f6d97fe996caeacafd04211dfa7aea" exitCode=6
Jul 17 20:41:39.871597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:39.871169    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" event={"ID":"8b9c0f01-f474-421a-b7c7-41aa93eecaed","Type":"ContainerDied","Data":"f500a06020daa831ab4f6df4eb497695c8f6d97fe996caeacafd04211dfa7aea"}
Jul 17 20:41:39.871597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:39.871221    2572 scope.go:117] "RemoveContainer" containerID="7141a450ba4b4363f06f3baaae0c2a1ba5e1b9975d133d07654cde8e92fbfaeb"
Jul 17 20:41:39.871597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:39.871444    2572 scope.go:117] "RemoveContainer" containerID="f500a06020daa831ab4f6df4eb497695c8f6d97fe996caeacafd04211dfa7aea"
Jul 17 20:41:39.871751 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:39.871635    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr_odh-ai-gateway-infra(8b9c0f01-f474-421a-b7c7-41aa93eecaed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed"
Jul 17 20:41:40.085637 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:40.085600    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:41:40.877099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:40.877075    2572 scope.go:117] "RemoveContainer" containerID="f500a06020daa831ab4f6df4eb497695c8f6d97fe996caeacafd04211dfa7aea"
Jul 17 20:41:40.877467 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:40.877230    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr_odh-ai-gateway-infra(8b9c0f01-f474-421a-b7c7-41aa93eecaed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed"
Jul 17 20:41:41.341059 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.341018    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"]
Jul 17 20:41:41.341332 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.341321    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:41.341388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.341333    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="18141782-2467-43f4-95e5-90f640e389b9" containerName="revoke-keys"
Jul 17 20:41:41.341388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.341344    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:41.341388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.341350    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:41.341499 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.341403    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ab9170-b00d-45db-919f-eea72c5254f8" containerName="revoke-keys"
Jul 17 20:41:41.345478 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.345463    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"
Jul 17 20:41:41.350239 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.350213    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"]
Jul 17 20:41:41.362477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.362449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0680be2a-01e7-40aa-a470-efbdaa22e369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-5jfp6\" (UID: \"0680be2a-01e7-40aa-a470-efbdaa22e369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"
Jul 17 20:41:41.463756 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.463725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0680be2a-01e7-40aa-a470-efbdaa22e369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-5jfp6\" (UID: \"0680be2a-01e7-40aa-a470-efbdaa22e369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"
Jul 17 20:41:41.464338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.464321    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0680be2a-01e7-40aa-a470-efbdaa22e369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-5jfp6\" (UID: \"0680be2a-01e7-40aa-a470-efbdaa22e369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"
Jul 17 20:41:41.656581 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.656499    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"
Jul 17 20:41:41.785706 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.785668    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"]
Jul 17 20:41:41.787769 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:41:41.787735    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0680be2a_01e7_40aa_a470_efbdaa22e369.slice/crio-93da8e7901d9ff7e08b0df385361c276bf9afa3645b16625f717b0cefd6308cd WatchSource:0}: Error finding container 93da8e7901d9ff7e08b0df385361c276bf9afa3645b16625f717b0cefd6308cd: Status 404 returned error can't find the container with id 93da8e7901d9ff7e08b0df385361c276bf9afa3645b16625f717b0cefd6308cd
Jul 17 20:41:41.882387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:41.882359    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" event={"ID":"0680be2a-01e7-40aa-a470-efbdaa22e369","Type":"ContainerStarted","Data":"93da8e7901d9ff7e08b0df385361c276bf9afa3645b16625f717b0cefd6308cd"}
Jul 17 20:41:42.888499 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:42.888465    2572 generic.go:356] "Generic (PLEG): container finished" podID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerID="62bf1a436975ec8c7d5ca90d6eb3aa94f6e915ea2b8e4610c6aeeadbc7a3af5c" exitCode=6
Jul 17 20:41:42.888876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:42.888552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" event={"ID":"0680be2a-01e7-40aa-a470-efbdaa22e369","Type":"ContainerDied","Data":"62bf1a436975ec8c7d5ca90d6eb3aa94f6e915ea2b8e4610c6aeeadbc7a3af5c"}
Jul 17 20:41:42.888876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:42.888730    2572 scope.go:117] "RemoveContainer" containerID="62bf1a436975ec8c7d5ca90d6eb3aa94f6e915ea2b8e4610c6aeeadbc7a3af5c"
Jul 17 20:41:43.894238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:43.894144    2572 generic.go:356] "Generic (PLEG): container finished" podID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerID="a457f4f9f9a37bb993c894cdbbc1397f89aaaba9bf97257003db21c7856ca1d4" exitCode=6
Jul 17 20:41:43.894238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:43.894228    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" event={"ID":"0680be2a-01e7-40aa-a470-efbdaa22e369","Type":"ContainerDied","Data":"a457f4f9f9a37bb993c894cdbbc1397f89aaaba9bf97257003db21c7856ca1d4"}
Jul 17 20:41:43.894722 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:43.894285    2572 scope.go:117] "RemoveContainer" containerID="62bf1a436975ec8c7d5ca90d6eb3aa94f6e915ea2b8e4610c6aeeadbc7a3af5c"
Jul 17 20:41:43.894722 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:43.894484    2572 scope.go:117] "RemoveContainer" containerID="a457f4f9f9a37bb993c894cdbbc1397f89aaaba9bf97257003db21c7856ca1d4"
Jul 17 20:41:43.894722 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:43.894678    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-5jfp6_odh-ai-gateway-infra(0680be2a-01e7-40aa-a470-efbdaa22e369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369"
Jul 17 20:41:44.899954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:44.899921    2572 scope.go:117] "RemoveContainer" containerID="a457f4f9f9a37bb993c894cdbbc1397f89aaaba9bf97257003db21c7856ca1d4"
Jul 17 20:41:44.900400 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:44.900129    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-5jfp6_odh-ai-gateway-infra(0680be2a-01e7-40aa-a470-efbdaa22e369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369"
Jul 17 20:41:49.425674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.425633    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"]
Jul 17 20:41:49.428918 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.428891    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"
Jul 17 20:41:49.438207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.438182    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"]
Jul 17 20:41:49.532574 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.532525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5493cbc6-9be4-49c6-ad77-6883a9773305-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9\" (UID: \"5493cbc6-9be4-49c6-ad77-6883a9773305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"
Jul 17 20:41:49.633203 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.633159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5493cbc6-9be4-49c6-ad77-6883a9773305-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9\" (UID: \"5493cbc6-9be4-49c6-ad77-6883a9773305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"
Jul 17 20:41:49.633795 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.633774    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5493cbc6-9be4-49c6-ad77-6883a9773305-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9\" (UID: \"5493cbc6-9be4-49c6-ad77-6883a9773305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"
Jul 17 20:41:49.741057 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.741002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"
Jul 17 20:41:49.868923 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.868885    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"]
Jul 17 20:41:49.875138 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:41:49.875106    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5493cbc6_9be4_49c6_ad77_6883a9773305.slice/crio-e24d1c7b47e15865330feb4fba77a3f830031ffd1251e95f9855cab55c6bc2c6 WatchSource:0}: Error finding container e24d1c7b47e15865330feb4fba77a3f830031ffd1251e95f9855cab55c6bc2c6: Status 404 returned error can't find the container with id e24d1c7b47e15865330feb4fba77a3f830031ffd1251e95f9855cab55c6bc2c6
Jul 17 20:41:49.922740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:49.922705    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" event={"ID":"5493cbc6-9be4-49c6-ad77-6883a9773305","Type":"ContainerStarted","Data":"e24d1c7b47e15865330feb4fba77a3f830031ffd1251e95f9855cab55c6bc2c6"}
Jul 17 20:41:50.930066 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:50.929969    2572 generic.go:356] "Generic (PLEG): container finished" podID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerID="43081bdda98c0581875bb52548f98bebada143e1b70cb1b26875a01d18a6357e" exitCode=6
Jul 17 20:41:50.930434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:50.930067    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" event={"ID":"5493cbc6-9be4-49c6-ad77-6883a9773305","Type":"ContainerDied","Data":"43081bdda98c0581875bb52548f98bebada143e1b70cb1b26875a01d18a6357e"}
Jul 17 20:41:50.930434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:50.930238    2572 scope.go:117] "RemoveContainer" containerID="43081bdda98c0581875bb52548f98bebada143e1b70cb1b26875a01d18a6357e"
Jul 17 20:41:51.936707 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:51.936602    2572 generic.go:356] "Generic (PLEG): container finished" podID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerID="9a4be484481294914b98b8ce26d418252269954e7498382150323d65a90a2f95" exitCode=6
Jul 17 20:41:51.936707 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:51.936675    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" event={"ID":"5493cbc6-9be4-49c6-ad77-6883a9773305","Type":"ContainerDied","Data":"9a4be484481294914b98b8ce26d418252269954e7498382150323d65a90a2f95"}
Jul 17 20:41:51.937226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:51.936727    2572 scope.go:117] "RemoveContainer" containerID="43081bdda98c0581875bb52548f98bebada143e1b70cb1b26875a01d18a6357e"
Jul 17 20:41:51.937226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:51.936896    2572 scope.go:117] "RemoveContainer" containerID="9a4be484481294914b98b8ce26d418252269954e7498382150323d65a90a2f95"
Jul 17 20:41:51.937226 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:51.937104    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9_odh-ai-gateway-infra(5493cbc6-9be4-49c6-ad77-6883a9773305)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305"
Jul 17 20:41:52.942534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:52.942506    2572 scope.go:117] "RemoveContainer" containerID="9a4be484481294914b98b8ce26d418252269954e7498382150323d65a90a2f95"
Jul 17 20:41:52.942944 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:52.942686    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9_odh-ai-gateway-infra(5493cbc6-9be4-49c6-ad77-6883a9773305)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305"
Jul 17 20:41:53.848343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:53.848304    2572 scope.go:117] "RemoveContainer" containerID="f500a06020daa831ab4f6df4eb497695c8f6d97fe996caeacafd04211dfa7aea"
Jul 17 20:41:54.599199 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:54.599158    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:41:54.953122 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:54.953089    2572 generic.go:356] "Generic (PLEG): container finished" podID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerID="a37e2707e1291cf4f60fafe3c77d24e13bb12d181a251a320564a978d4b1fe55" exitCode=6
Jul 17 20:41:54.953303 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:54.953161    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" event={"ID":"8b9c0f01-f474-421a-b7c7-41aa93eecaed","Type":"ContainerDied","Data":"a37e2707e1291cf4f60fafe3c77d24e13bb12d181a251a320564a978d4b1fe55"}
Jul 17 20:41:54.953303 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:54.953224    2572 scope.go:117] "RemoveContainer" containerID="f500a06020daa831ab4f6df4eb497695c8f6d97fe996caeacafd04211dfa7aea"
Jul 17 20:41:54.953507 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:54.953492    2572 scope.go:117] "RemoveContainer" containerID="a37e2707e1291cf4f60fafe3c77d24e13bb12d181a251a320564a978d4b1fe55"
Jul 17 20:41:54.953696 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:54.953676    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr_odh-ai-gateway-infra(8b9c0f01-f474-421a-b7c7-41aa93eecaed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed"
Jul 17 20:41:55.980333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:55.980286    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"]
Jul 17 20:41:56.118821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.118795    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"
Jul 17 20:41:56.190066 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.190020    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9c0f01-f474-421a-b7c7-41aa93eecaed-maas-api-service-ca\") pod \"8b9c0f01-f474-421a-b7c7-41aa93eecaed\" (UID: \"8b9c0f01-f474-421a-b7c7-41aa93eecaed\") "
Jul 17 20:41:56.190414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.190387    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9c0f01-f474-421a-b7c7-41aa93eecaed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b9c0f01-f474-421a-b7c7-41aa93eecaed" (UID: "8b9c0f01-f474-421a-b7c7-41aa93eecaed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:41:56.290888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.290812    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9c0f01-f474-421a-b7c7-41aa93eecaed-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:41:56.848714 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.848679    2572 scope.go:117] "RemoveContainer" containerID="a457f4f9f9a37bb993c894cdbbc1397f89aaaba9bf97257003db21c7856ca1d4"
Jul 17 20:41:56.964435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.964404    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr" event={"ID":"8b9c0f01-f474-421a-b7c7-41aa93eecaed","Type":"ContainerDied","Data":"e3cd8507bfd0c63b59cdd347ccfc315b75d9b207a325c3e4bdb739e5ca5206a2"}
Jul 17 20:41:56.964435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.964437    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"
Jul 17 20:41:56.964637 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.964462    2572 scope.go:117] "RemoveContainer" containerID="a37e2707e1291cf4f60fafe3c77d24e13bb12d181a251a320564a978d4b1fe55"
Jul 17 20:41:56.994006 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:56.993976    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"]
Jul 17 20:41:57.003102 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:57.003069    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-lhcbr"]
Jul 17 20:41:57.853990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:57.853950    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" path="/var/lib/kubelet/pods/8b9c0f01-f474-421a-b7c7-41aa93eecaed/volumes"
Jul 17 20:41:57.972099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:57.972042    2572 generic.go:356] "Generic (PLEG): container finished" podID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerID="18f2431e6c637eaa1185f8bb4a5950f0f3f6587835ed9aff2c422c18271d6045" exitCode=6
Jul 17 20:41:57.972280 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:57.972114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" event={"ID":"0680be2a-01e7-40aa-a470-efbdaa22e369","Type":"ContainerDied","Data":"18f2431e6c637eaa1185f8bb4a5950f0f3f6587835ed9aff2c422c18271d6045"}
Jul 17 20:41:57.972280 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:57.972169    2572 scope.go:117] "RemoveContainer" containerID="a457f4f9f9a37bb993c894cdbbc1397f89aaaba9bf97257003db21c7856ca1d4"
Jul 17 20:41:57.972441 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:57.972425    2572 scope.go:117] "RemoveContainer" containerID="18f2431e6c637eaa1185f8bb4a5950f0f3f6587835ed9aff2c422c18271d6045"
Jul 17 20:41:57.972660 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:41:57.972641    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-5jfp6_odh-ai-gateway-infra(0680be2a-01e7-40aa-a470-efbdaa22e369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369"
Jul 17 20:41:58.998150 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:58.998121    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"]
Jul 17 20:41:59.131601 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.131570    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"
Jul 17 20:41:59.214232 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.214203    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0680be2a-01e7-40aa-a470-efbdaa22e369-maas-api-service-ca\") pod \"0680be2a-01e7-40aa-a470-efbdaa22e369\" (UID: \"0680be2a-01e7-40aa-a470-efbdaa22e369\") "
Jul 17 20:41:59.214568 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.214536    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0680be2a-01e7-40aa-a470-efbdaa22e369-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0680be2a-01e7-40aa-a470-efbdaa22e369" (UID: "0680be2a-01e7-40aa-a470-efbdaa22e369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:41:59.314744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.314657    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0680be2a-01e7-40aa-a470-efbdaa22e369-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:41:59.983678 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.983654    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"
Jul 17 20:41:59.983853 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.983690    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6" event={"ID":"0680be2a-01e7-40aa-a470-efbdaa22e369","Type":"ContainerDied","Data":"93da8e7901d9ff7e08b0df385361c276bf9afa3645b16625f717b0cefd6308cd"}
Jul 17 20:41:59.983853 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.983748    2572 scope.go:117] "RemoveContainer" containerID="18f2431e6c637eaa1185f8bb4a5950f0f3f6587835ed9aff2c422c18271d6045"
Jul 17 20:41:59.999293 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:41:59.999267    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"]
Jul 17 20:42:00.002959 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:00.002939    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-5jfp6"]
Jul 17 20:42:01.852585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:01.852553    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" path="/var/lib/kubelet/pods/0680be2a-01e7-40aa-a470-efbdaa22e369/volumes"
Jul 17 20:42:05.366579 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.366533    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"]
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.366974    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.366993    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367006    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367015    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367025    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367036    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367068    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367077    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367099    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367108    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367186    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367197    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:05.367789 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367208    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:05.367789 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.367217    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:05.371731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.371712    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"
Jul 17 20:42:05.377349 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.377326    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"]
Jul 17 20:42:05.462990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.462950    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d5a98-41e8-421b-8671-041c79116a07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb\" (UID: \"4b9d5a98-41e8-421b-8671-041c79116a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"
Jul 17 20:42:05.564206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.564168    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d5a98-41e8-421b-8671-041c79116a07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb\" (UID: \"4b9d5a98-41e8-421b-8671-041c79116a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"
Jul 17 20:42:05.564792 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.564771    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d5a98-41e8-421b-8671-041c79116a07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb\" (UID: \"4b9d5a98-41e8-421b-8671-041c79116a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"
Jul 17 20:42:05.683032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.682949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"
Jul 17 20:42:05.812963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:05.812939    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"]
Jul 17 20:42:05.814791 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:42:05.814767    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9d5a98_41e8_421b_8671_041c79116a07.slice/crio-2974acd3fd734d63fb850ed39258fc556a6179f4012929b6ecf5cdd6691fb12a WatchSource:0}: Error finding container 2974acd3fd734d63fb850ed39258fc556a6179f4012929b6ecf5cdd6691fb12a: Status 404 returned error can't find the container with id 2974acd3fd734d63fb850ed39258fc556a6179f4012929b6ecf5cdd6691fb12a
Jul 17 20:42:06.011436 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:06.011395    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" event={"ID":"4b9d5a98-41e8-421b-8671-041c79116a07","Type":"ContainerStarted","Data":"2974acd3fd734d63fb850ed39258fc556a6179f4012929b6ecf5cdd6691fb12a"}
Jul 17 20:42:06.848542 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:06.848516    2572 scope.go:117] "RemoveContainer" containerID="9a4be484481294914b98b8ce26d418252269954e7498382150323d65a90a2f95"
Jul 17 20:42:07.016723 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:07.016689    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b9d5a98-41e8-421b-8671-041c79116a07" containerID="9d257395181f6de51fbf330dacd17cc15587d5e106dc987cb8ec88e7aadb3cfd" exitCode=6
Jul 17 20:42:07.016902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:07.016779    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" event={"ID":"4b9d5a98-41e8-421b-8671-041c79116a07","Type":"ContainerDied","Data":"9d257395181f6de51fbf330dacd17cc15587d5e106dc987cb8ec88e7aadb3cfd"}
Jul 17 20:42:07.016996 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:07.016972    2572 scope.go:117] "RemoveContainer" containerID="9d257395181f6de51fbf330dacd17cc15587d5e106dc987cb8ec88e7aadb3cfd"
Jul 17 20:42:08.022866 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.022832    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b9d5a98-41e8-421b-8671-041c79116a07" containerID="b6fcb70c0a38c23342fd4c8e4318f63550f9f94e64222e97daeb79f72eb33ab9" exitCode=6
Jul 17 20:42:08.023357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.022896    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" event={"ID":"4b9d5a98-41e8-421b-8671-041c79116a07","Type":"ContainerDied","Data":"b6fcb70c0a38c23342fd4c8e4318f63550f9f94e64222e97daeb79f72eb33ab9"}
Jul 17 20:42:08.023357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.022960    2572 scope.go:117] "RemoveContainer" containerID="9d257395181f6de51fbf330dacd17cc15587d5e106dc987cb8ec88e7aadb3cfd"
Jul 17 20:42:08.023357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.023207    2572 scope.go:117] "RemoveContainer" containerID="b6fcb70c0a38c23342fd4c8e4318f63550f9f94e64222e97daeb79f72eb33ab9"
Jul 17 20:42:08.023492 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:08.023379    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb_odh-ai-gateway-infra(4b9d5a98-41e8-421b-8671-041c79116a07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" podUID="4b9d5a98-41e8-421b-8671-041c79116a07"
Jul 17 20:42:08.024713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.024694    2572 generic.go:356] "Generic (PLEG): container finished" podID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerID="b246be6d3a726c44f2322cd603cd3699432f7c25c6638c97cab4de8a4771f5da" exitCode=6
Jul 17 20:42:08.024808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.024757    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" event={"ID":"5493cbc6-9be4-49c6-ad77-6883a9773305","Type":"ContainerDied","Data":"b246be6d3a726c44f2322cd603cd3699432f7c25c6638c97cab4de8a4771f5da"}
Jul 17 20:42:08.024998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.024986    2572 scope.go:117] "RemoveContainer" containerID="b246be6d3a726c44f2322cd603cd3699432f7c25c6638c97cab4de8a4771f5da"
Jul 17 20:42:08.025159 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:08.025145    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9_odh-ai-gateway-infra(5493cbc6-9be4-49c6-ad77-6883a9773305)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305"
Jul 17 20:42:08.033350 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:08.033335    2572 scope.go:117] "RemoveContainer" containerID="9a4be484481294914b98b8ce26d418252269954e7498382150323d65a90a2f95"
Jul 17 20:42:09.030679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:09.030654    2572 scope.go:117] "RemoveContainer" containerID="b6fcb70c0a38c23342fd4c8e4318f63550f9f94e64222e97daeb79f72eb33ab9"
Jul 17 20:42:09.031184 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:09.030874    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb_odh-ai-gateway-infra(4b9d5a98-41e8-421b-8671-041c79116a07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" podUID="4b9d5a98-41e8-421b-8671-041c79116a07"
Jul 17 20:42:09.061083 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:09.061038    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"]
Jul 17 20:42:09.195876 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:09.195855    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"
Jul 17 20:42:09.294645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:09.294555    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5493cbc6-9be4-49c6-ad77-6883a9773305-maas-api-service-ca\") pod \"5493cbc6-9be4-49c6-ad77-6883a9773305\" (UID: \"5493cbc6-9be4-49c6-ad77-6883a9773305\") "
Jul 17 20:42:09.294911 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:09.294886    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5493cbc6-9be4-49c6-ad77-6883a9773305-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5493cbc6-9be4-49c6-ad77-6883a9773305" (UID: "5493cbc6-9be4-49c6-ad77-6883a9773305"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:42:09.395532 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:09.395496    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5493cbc6-9be4-49c6-ad77-6883a9773305-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:42:10.037076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:10.037026    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9" event={"ID":"5493cbc6-9be4-49c6-ad77-6883a9773305","Type":"ContainerDied","Data":"e24d1c7b47e15865330feb4fba77a3f830031ffd1251e95f9855cab55c6bc2c6"}
Jul 17 20:42:10.037076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:10.037075    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"
Jul 17 20:42:10.037577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:10.037103    2572 scope.go:117] "RemoveContainer" containerID="b246be6d3a726c44f2322cd603cd3699432f7c25c6638c97cab4de8a4771f5da"
Jul 17 20:42:10.055581 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:10.055556    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"]
Jul 17 20:42:10.059821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:10.059800    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-b8ch9"]
Jul 17 20:42:11.855616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:11.855574    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" path="/var/lib/kubelet/pods/5493cbc6-9be4-49c6-ad77-6883a9773305/volumes"
Jul 17 20:42:23.848015 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:23.847892    2572 scope.go:117] "RemoveContainer" containerID="b6fcb70c0a38c23342fd4c8e4318f63550f9f94e64222e97daeb79f72eb33ab9"
Jul 17 20:42:25.105407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:25.105373    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b9d5a98-41e8-421b-8671-041c79116a07" containerID="fa24cc82a509dd59b76b3527452266d77d516f959aafa7228498266a18d07273" exitCode=6
Jul 17 20:42:25.105874 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:25.105457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" event={"ID":"4b9d5a98-41e8-421b-8671-041c79116a07","Type":"ContainerDied","Data":"fa24cc82a509dd59b76b3527452266d77d516f959aafa7228498266a18d07273"}
Jul 17 20:42:25.105874 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:25.105539    2572 scope.go:117] "RemoveContainer" containerID="b6fcb70c0a38c23342fd4c8e4318f63550f9f94e64222e97daeb79f72eb33ab9"
Jul 17 20:42:25.105874 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:25.105852    2572 scope.go:117] "RemoveContainer" containerID="fa24cc82a509dd59b76b3527452266d77d516f959aafa7228498266a18d07273"
Jul 17 20:42:25.106111 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:25.106089    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb_odh-ai-gateway-infra(4b9d5a98-41e8-421b-8671-041c79116a07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" podUID="4b9d5a98-41e8-421b-8671-041c79116a07"
Jul 17 20:42:26.143275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:26.143238    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"]
Jul 17 20:42:26.275395 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:26.275373    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"
Jul 17 20:42:26.434686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:26.434595    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d5a98-41e8-421b-8671-041c79116a07-maas-api-service-ca\") pod \"4b9d5a98-41e8-421b-8671-041c79116a07\" (UID: \"4b9d5a98-41e8-421b-8671-041c79116a07\") "
Jul 17 20:42:26.434949 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:26.434924    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9d5a98-41e8-421b-8671-041c79116a07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b9d5a98-41e8-421b-8671-041c79116a07" (UID: "4b9d5a98-41e8-421b-8671-041c79116a07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:42:26.535761 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:26.535729    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d5a98-41e8-421b-8671-041c79116a07-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:42:27.117012 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.116984    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"
Jul 17 20:42:27.117244 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.116975    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb" event={"ID":"4b9d5a98-41e8-421b-8671-041c79116a07","Type":"ContainerDied","Data":"2974acd3fd734d63fb850ed39258fc556a6179f4012929b6ecf5cdd6691fb12a"}
Jul 17 20:42:27.117244 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.117120    2572 scope.go:117] "RemoveContainer" containerID="fa24cc82a509dd59b76b3527452266d77d516f959aafa7228498266a18d07273"
Jul 17 20:42:27.144954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.141434    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"]
Jul 17 20:42:27.147220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.147193    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m8rkb"]
Jul 17 20:42:27.613597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613563    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"]
Jul 17 20:42:27.613896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613884    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:27.613944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613897    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:27.613944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613907    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.613944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613912    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.613944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613919    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.613944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613925    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.613944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613936    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.613944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613941    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613948    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613953    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613958    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.613964    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614012    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614019    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614025    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0680be2a-01e7-40aa-a470-efbdaa22e369" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614031    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614036    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614040    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9c0f01-f474-421a-b7c7-41aa93eecaed" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614059    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:27.614175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.614065    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:27.618126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.618108    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"
Jul 17 20:42:27.620881 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.620861    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:42:27.620998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.620862    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:42:27.628709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.628685    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"]
Jul 17 20:42:27.746638 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.746600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a246af9d-542e-49b0-ba21-e164dbc5a5b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j\" (UID: \"a246af9d-542e-49b0-ba21-e164dbc5a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"
Jul 17 20:42:27.847281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.847240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a246af9d-542e-49b0-ba21-e164dbc5a5b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j\" (UID: \"a246af9d-542e-49b0-ba21-e164dbc5a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"
Jul 17 20:42:27.847928 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.847899    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a246af9d-542e-49b0-ba21-e164dbc5a5b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j\" (UID: \"a246af9d-542e-49b0-ba21-e164dbc5a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"
Jul 17 20:42:27.852736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.852709    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9d5a98-41e8-421b-8671-041c79116a07" path="/var/lib/kubelet/pods/4b9d5a98-41e8-421b-8671-041c79116a07/volumes"
Jul 17 20:42:27.927982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:27.927890    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"
Jul 17 20:42:28.078418 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:28.078391    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"]
Jul 17 20:42:28.080570 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:42:28.080543    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda246af9d_542e_49b0_ba21_e164dbc5a5b1.slice/crio-893fed94ffe19fd89730b70bb1324abf8604706d2f056bc0f02015411392aab3 WatchSource:0}: Error finding container 893fed94ffe19fd89730b70bb1324abf8604706d2f056bc0f02015411392aab3: Status 404 returned error can't find the container with id 893fed94ffe19fd89730b70bb1324abf8604706d2f056bc0f02015411392aab3
Jul 17 20:42:28.123391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:28.123353    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" event={"ID":"a246af9d-542e-49b0-ba21-e164dbc5a5b1","Type":"ContainerStarted","Data":"893fed94ffe19fd89730b70bb1324abf8604706d2f056bc0f02015411392aab3"}
Jul 17 20:42:29.129037 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:29.129002    2572 generic.go:356] "Generic (PLEG): container finished" podID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerID="fc79d6788d97510e112b2c5e38642cbf25708029fed3605046f34c8992fe6746" exitCode=6
Jul 17 20:42:29.129460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:29.129094    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" event={"ID":"a246af9d-542e-49b0-ba21-e164dbc5a5b1","Type":"ContainerDied","Data":"fc79d6788d97510e112b2c5e38642cbf25708029fed3605046f34c8992fe6746"}
Jul 17 20:42:29.129460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:29.129289    2572 scope.go:117] "RemoveContainer" containerID="fc79d6788d97510e112b2c5e38642cbf25708029fed3605046f34c8992fe6746"
Jul 17 20:42:30.136037 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:30.135948    2572 generic.go:356] "Generic (PLEG): container finished" podID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerID="60d317fdba19c27f29b41e6c267372fbafbc03c1189c3d45ad9e471e9d12e01b" exitCode=6
Jul 17 20:42:30.136037 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:30.135983    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" event={"ID":"a246af9d-542e-49b0-ba21-e164dbc5a5b1","Type":"ContainerDied","Data":"60d317fdba19c27f29b41e6c267372fbafbc03c1189c3d45ad9e471e9d12e01b"}
Jul 17 20:42:30.136037 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:30.136028    2572 scope.go:117] "RemoveContainer" containerID="fc79d6788d97510e112b2c5e38642cbf25708029fed3605046f34c8992fe6746"
Jul 17 20:42:30.136548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:30.136281    2572 scope.go:117] "RemoveContainer" containerID="60d317fdba19c27f29b41e6c267372fbafbc03c1189c3d45ad9e471e9d12e01b"
Jul 17 20:42:30.136548 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:30.136504    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j_odh-ai-gateway-infra(a246af9d-542e-49b0-ba21-e164dbc5a5b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1"
Jul 17 20:42:31.141935 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.141908    2572 scope.go:117] "RemoveContainer" containerID="60d317fdba19c27f29b41e6c267372fbafbc03c1189c3d45ad9e471e9d12e01b"
Jul 17 20:42:31.142400 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:31.142178    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j_odh-ai-gateway-infra(a246af9d-542e-49b0-ba21-e164dbc5a5b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1"
Jul 17 20:42:31.385745 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.385704    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"]
Jul 17 20:42:31.386079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.386058    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:31.386079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.386073    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5493cbc6-9be4-49c6-ad77-6883a9773305" containerName="revoke-keys"
Jul 17 20:42:31.390286 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.390265    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"
Jul 17 20:42:31.396099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.396012    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"]
Jul 17 20:42:31.576028 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.575992    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d664825b-4264-45ba-9b75-90be8e2fd108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-29rqt\" (UID: \"d664825b-4264-45ba-9b75-90be8e2fd108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"
Jul 17 20:42:31.677227 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.677131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d664825b-4264-45ba-9b75-90be8e2fd108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-29rqt\" (UID: \"d664825b-4264-45ba-9b75-90be8e2fd108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"
Jul 17 20:42:31.677816 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.677795    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d664825b-4264-45ba-9b75-90be8e2fd108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-29rqt\" (UID: \"d664825b-4264-45ba-9b75-90be8e2fd108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"
Jul 17 20:42:31.702774 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.702741    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"
Jul 17 20:42:31.834748 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:31.834658    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"]
Jul 17 20:42:31.836779 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:42:31.836752    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd664825b_4264_45ba_9b75_90be8e2fd108.slice/crio-2b30b8e1464e7207489471c262dd63b48f971bc2b41514283a5364774eec9f19 WatchSource:0}: Error finding container 2b30b8e1464e7207489471c262dd63b48f971bc2b41514283a5364774eec9f19: Status 404 returned error can't find the container with id 2b30b8e1464e7207489471c262dd63b48f971bc2b41514283a5364774eec9f19
Jul 17 20:42:32.147079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:32.147030    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" event={"ID":"d664825b-4264-45ba-9b75-90be8e2fd108","Type":"ContainerStarted","Data":"f23ad4f77752e8529fae09a9fa8367dea26dd5cf653f12d614e09a10f66dcbfd"}
Jul 17 20:42:32.147444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:32.147104    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" event={"ID":"d664825b-4264-45ba-9b75-90be8e2fd108","Type":"ContainerStarted","Data":"2b30b8e1464e7207489471c262dd63b48f971bc2b41514283a5364774eec9f19"}
Jul 17 20:42:33.153453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:33.153419    2572 generic.go:356] "Generic (PLEG): container finished" podID="d664825b-4264-45ba-9b75-90be8e2fd108" containerID="f23ad4f77752e8529fae09a9fa8367dea26dd5cf653f12d614e09a10f66dcbfd" exitCode=6
Jul 17 20:42:33.153847 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:33.153479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" event={"ID":"d664825b-4264-45ba-9b75-90be8e2fd108","Type":"ContainerDied","Data":"f23ad4f77752e8529fae09a9fa8367dea26dd5cf653f12d614e09a10f66dcbfd"}
Jul 17 20:42:33.153847 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:33.153682    2572 scope.go:117] "RemoveContainer" containerID="f23ad4f77752e8529fae09a9fa8367dea26dd5cf653f12d614e09a10f66dcbfd"
Jul 17 20:42:34.160342 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:34.160245    2572 generic.go:356] "Generic (PLEG): container finished" podID="d664825b-4264-45ba-9b75-90be8e2fd108" containerID="f16c2e5f81df0230a947607b19d0baba41097276ee938b696bf9696d89d013e4" exitCode=6
Jul 17 20:42:34.160342 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:34.160313    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" event={"ID":"d664825b-4264-45ba-9b75-90be8e2fd108","Type":"ContainerDied","Data":"f16c2e5f81df0230a947607b19d0baba41097276ee938b696bf9696d89d013e4"}
Jul 17 20:42:34.160778 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:34.160367    2572 scope.go:117] "RemoveContainer" containerID="f23ad4f77752e8529fae09a9fa8367dea26dd5cf653f12d614e09a10f66dcbfd"
Jul 17 20:42:34.160778 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:34.160612    2572 scope.go:117] "RemoveContainer" containerID="f16c2e5f81df0230a947607b19d0baba41097276ee938b696bf9696d89d013e4"
Jul 17 20:42:34.160867 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:34.160838    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-29rqt_odh-ai-gateway-infra(d664825b-4264-45ba-9b75-90be8e2fd108)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" podUID="d664825b-4264-45ba-9b75-90be8e2fd108"
Jul 17 20:42:35.166345 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:35.166306    2572 scope.go:117] "RemoveContainer" containerID="f16c2e5f81df0230a947607b19d0baba41097276ee938b696bf9696d89d013e4"
Jul 17 20:42:35.166745 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:35.166480    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-29rqt_odh-ai-gateway-infra(d664825b-4264-45ba-9b75-90be8e2fd108)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" podUID="d664825b-4264-45ba-9b75-90be8e2fd108"
Jul 17 20:42:42.289913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.289873    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:42:42.681182 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.681113    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"]
Jul 17 20:42:42.685587 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.685568    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.690201 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.690175    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mxjrb\""
Jul 17 20:42:42.690323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.690175    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 20:42:42.690901 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.690880    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5f0f4f-serving-cert\""
Jul 17 20:42:42.705857 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.705836    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"]
Jul 17 20:42:42.756483 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.756452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.756677 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.756489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15470ed-8a04-4819-a609-0cf6f6472739-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.756677 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.756527    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b844\" (UniqueName: \"kubernetes.io/projected/c15470ed-8a04-4819-a609-0cf6f6472739-kube-api-access-6b844\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.848407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.848374    2572 scope.go:117] "RemoveContainer" containerID="60d317fdba19c27f29b41e6c267372fbafbc03c1189c3d45ad9e471e9d12e01b"
Jul 17 20:42:42.857364 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.857337    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15470ed-8a04-4819-a609-0cf6f6472739-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.857499 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.857408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b844\" (UniqueName: \"kubernetes.io/projected/c15470ed-8a04-4819-a609-0cf6f6472739-kube-api-access-6b844\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.857639 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.857501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.857639 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:42.857611    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-serving-cert: secret "maas-api-e2e-inf-b-5f0f4f-serving-cert" not found
Jul 17 20:42:42.857753 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:42.857677    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls podName:c15470ed-8a04-4819-a609-0cf6f6472739 nodeName:}" failed. No retries permitted until 2026-07-17 20:42:43.357655924 +0000 UTC m=+2798.051632068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls") pod "maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98" (UID: "c15470ed-8a04-4819-a609-0cf6f6472739") : secret "maas-api-e2e-inf-b-5f0f4f-serving-cert" not found
Jul 17 20:42:42.857968 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.857945    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15470ed-8a04-4819-a609-0cf6f6472739-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:42.874873 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:42.874846    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b844\" (UniqueName: \"kubernetes.io/projected/c15470ed-8a04-4819-a609-0cf6f6472739-kube-api-access-6b844\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:43.205062 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:43.205025    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" event={"ID":"a246af9d-542e-49b0-ba21-e164dbc5a5b1","Type":"ContainerStarted","Data":"4c63001c17f4698bbb0031f323751a7bcb76520d4732aa96c995e773b3301b39"}
Jul 17 20:42:43.228510 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:43.228450    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" podStartSLOduration=16.228430134 podStartE2EDuration="16.228430134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:42:43.226941658 +0000 UTC m=+2797.920917804" watchObservedRunningTime="2026-07-17 20:42:43.228430134 +0000 UTC m=+2797.922406280"
Jul 17 20:42:43.361694 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:43.361613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:43.363935 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:43.363914    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:43.597696 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:43.597662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:43.729323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:43.729217    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"]
Jul 17 20:42:43.731926 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:42:43.731901    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15470ed_8a04_4819_a609_0cf6f6472739.slice/crio-9427e4909204ed412ca9d89eca7701801812238ac1ff877e7532810e04a3b58b WatchSource:0}: Error finding container 9427e4909204ed412ca9d89eca7701801812238ac1ff877e7532810e04a3b58b: Status 404 returned error can't find the container with id 9427e4909204ed412ca9d89eca7701801812238ac1ff877e7532810e04a3b58b
Jul 17 20:42:43.733345 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:43.733329    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 20:42:44.210990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:44.210953    2572 generic.go:356] "Generic (PLEG): container finished" podID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerID="4c63001c17f4698bbb0031f323751a7bcb76520d4732aa96c995e773b3301b39" exitCode=6
Jul 17 20:42:44.211185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:44.211025    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" event={"ID":"a246af9d-542e-49b0-ba21-e164dbc5a5b1","Type":"ContainerDied","Data":"4c63001c17f4698bbb0031f323751a7bcb76520d4732aa96c995e773b3301b39"}
Jul 17 20:42:44.211185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:44.211103    2572 scope.go:117] "RemoveContainer" containerID="60d317fdba19c27f29b41e6c267372fbafbc03c1189c3d45ad9e471e9d12e01b"
Jul 17 20:42:44.211388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:44.211369    2572 scope.go:117] "RemoveContainer" containerID="4c63001c17f4698bbb0031f323751a7bcb76520d4732aa96c995e773b3301b39"
Jul 17 20:42:44.211633 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:44.211597    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j_odh-ai-gateway-infra(a246af9d-542e-49b0-ba21-e164dbc5a5b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1"
Jul 17 20:42:44.212369 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:44.212283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98" event={"ID":"c15470ed-8a04-4819-a609-0cf6f6472739","Type":"ContainerStarted","Data":"9427e4909204ed412ca9d89eca7701801812238ac1ff877e7532810e04a3b58b"}
Jul 17 20:42:44.236651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:44.236625    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"]
Jul 17 20:42:46.168649 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.168623    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"
Jul 17 20:42:46.183868 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.183833    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a246af9d-542e-49b0-ba21-e164dbc5a5b1-maas-api-service-ca\") pod \"a246af9d-542e-49b0-ba21-e164dbc5a5b1\" (UID: \"a246af9d-542e-49b0-ba21-e164dbc5a5b1\") "
Jul 17 20:42:46.184190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.184170    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a246af9d-542e-49b0-ba21-e164dbc5a5b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a246af9d-542e-49b0-ba21-e164dbc5a5b1" (UID: "a246af9d-542e-49b0-ba21-e164dbc5a5b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:42:46.225175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.225149    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"
Jul 17 20:42:46.225175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.225164    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j" event={"ID":"a246af9d-542e-49b0-ba21-e164dbc5a5b1","Type":"ContainerDied","Data":"893fed94ffe19fd89730b70bb1324abf8604706d2f056bc0f02015411392aab3"}
Jul 17 20:42:46.225361 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.225228    2572 scope.go:117] "RemoveContainer" containerID="4c63001c17f4698bbb0031f323751a7bcb76520d4732aa96c995e773b3301b39"
Jul 17 20:42:46.257152 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.257128    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"]
Jul 17 20:42:46.264726 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.264700    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-mdt8j"]
Jul 17 20:42:46.284329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:46.284307    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a246af9d-542e-49b0-ba21-e164dbc5a5b1-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:42:47.231267 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:47.231225    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98" event={"ID":"c15470ed-8a04-4819-a609-0cf6f6472739","Type":"ContainerStarted","Data":"a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76"}
Jul 17 20:42:47.231733 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:47.231356    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:47.255746 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:47.255699    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98" podStartSLOduration=2.786268685 podStartE2EDuration="5.25568213s" totalImagesPullingTime="2.469413445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:42:47.253935101 +0000 UTC m=+2801.947911247" watchObservedRunningTime="2026-07-17 20:42:47.25568213 +0000 UTC m=+2801.949658275"
Jul 17 20:42:47.848701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:47.848672    2572 scope.go:117] "RemoveContainer" containerID="f16c2e5f81df0230a947607b19d0baba41097276ee938b696bf9696d89d013e4"
Jul 17 20:42:47.855290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:47.855263    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" path="/var/lib/kubelet/pods/a246af9d-542e-49b0-ba21-e164dbc5a5b1/volumes"
Jul 17 20:42:48.238949 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:48.238913    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" event={"ID":"d664825b-4264-45ba-9b75-90be8e2fd108","Type":"ContainerStarted","Data":"adf7115f29b22cef383edc3596d8cce92e9ba2b39578d77c6c6c0e22d9bc9f7c"}
Jul 17 20:42:48.260063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:48.260005    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" podStartSLOduration=17.259986597 podStartE2EDuration="17.259986597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:42:32.17100991 +0000 UTC m=+2786.864986056" watchObservedRunningTime="2026-07-17 20:42:48.259986597 +0000 UTC m=+2802.953962742"
Jul 17 20:42:49.245423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.245393    2572 generic.go:356] "Generic (PLEG): container finished" podID="d664825b-4264-45ba-9b75-90be8e2fd108" containerID="adf7115f29b22cef383edc3596d8cce92e9ba2b39578d77c6c6c0e22d9bc9f7c" exitCode=6
Jul 17 20:42:49.245792 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.245470    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" event={"ID":"d664825b-4264-45ba-9b75-90be8e2fd108","Type":"ContainerDied","Data":"adf7115f29b22cef383edc3596d8cce92e9ba2b39578d77c6c6c0e22d9bc9f7c"}
Jul 17 20:42:49.245792 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.245529    2572 scope.go:117] "RemoveContainer" containerID="f16c2e5f81df0230a947607b19d0baba41097276ee938b696bf9696d89d013e4"
Jul 17 20:42:49.245897 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.245801    2572 scope.go:117] "RemoveContainer" containerID="adf7115f29b22cef383edc3596d8cce92e9ba2b39578d77c6c6c0e22d9bc9f7c"
Jul 17 20:42:49.246005 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:49.245986    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-29rqt_odh-ai-gateway-infra(d664825b-4264-45ba-9b75-90be8e2fd108)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" podUID="d664825b-4264-45ba-9b75-90be8e2fd108"
Jul 17 20:42:49.275572 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.275544    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"]
Jul 17 20:42:49.477290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477258    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"]
Jul 17 20:42:49.477581 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477569    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.477627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477582    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.477627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477594    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.477627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477600    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.477627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477607    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.477627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477612    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.477780 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477670    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.477780 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.477677    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:49.481776 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.481757    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"
Jul 17 20:42:49.490558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.490522    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"]
Jul 17 20:42:49.510736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.510676    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8412454b-ac97-4060-aa42-d8a246555c5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr\" (UID: \"8412454b-ac97-4060-aa42-d8a246555c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"
Jul 17 20:42:49.612061 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.612017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8412454b-ac97-4060-aa42-d8a246555c5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr\" (UID: \"8412454b-ac97-4060-aa42-d8a246555c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"
Jul 17 20:42:49.612665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.612645    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8412454b-ac97-4060-aa42-d8a246555c5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr\" (UID: \"8412454b-ac97-4060-aa42-d8a246555c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"
Jul 17 20:42:49.797114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.797010    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"
Jul 17 20:42:49.949185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:49.949109    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"]
Jul 17 20:42:49.952228 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:42:49.952182    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8412454b_ac97_4060_aa42_d8a246555c5a.slice/crio-ee9755111b4b1e93f998cb5af94001b9f97ca7dbd72509e226e303890f9d8176 WatchSource:0}: Error finding container ee9755111b4b1e93f998cb5af94001b9f97ca7dbd72509e226e303890f9d8176: Status 404 returned error can't find the container with id ee9755111b4b1e93f998cb5af94001b9f97ca7dbd72509e226e303890f9d8176
Jul 17 20:42:50.253548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:50.253519    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" event={"ID":"8412454b-ac97-4060-aa42-d8a246555c5a","Type":"ContainerStarted","Data":"506ce444d9cdf9dba4bb94245da239971d3fdebedb46143456a0dbc70cae302d"}
Jul 17 20:42:50.253930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:50.253573    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" event={"ID":"8412454b-ac97-4060-aa42-d8a246555c5a","Type":"ContainerStarted","Data":"ee9755111b4b1e93f998cb5af94001b9f97ca7dbd72509e226e303890f9d8176"}
Jul 17 20:42:50.291145 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:50.291089    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" podStartSLOduration=1.2910728169999999 podStartE2EDuration="1.291072817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:42:50.288343431 +0000 UTC m=+2804.982319577" watchObservedRunningTime="2026-07-17 20:42:50.291072817 +0000 UTC m=+2804.985048965"
Jul 17 20:42:50.378840 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:50.378818    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"
Jul 17 20:42:50.418945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:50.418877    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d664825b-4264-45ba-9b75-90be8e2fd108-maas-api-service-ca\") pod \"d664825b-4264-45ba-9b75-90be8e2fd108\" (UID: \"d664825b-4264-45ba-9b75-90be8e2fd108\") "
Jul 17 20:42:50.419221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:50.419200    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d664825b-4264-45ba-9b75-90be8e2fd108-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d664825b-4264-45ba-9b75-90be8e2fd108" (UID: "d664825b-4264-45ba-9b75-90be8e2fd108"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:42:50.520394 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:50.520356    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d664825b-4264-45ba-9b75-90be8e2fd108-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:42:51.259341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.259311    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"
Jul 17 20:42:51.259778 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.259342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt" event={"ID":"d664825b-4264-45ba-9b75-90be8e2fd108","Type":"ContainerDied","Data":"2b30b8e1464e7207489471c262dd63b48f971bc2b41514283a5364774eec9f19"}
Jul 17 20:42:51.259778 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.259403    2572 scope.go:117] "RemoveContainer" containerID="adf7115f29b22cef383edc3596d8cce92e9ba2b39578d77c6c6c0e22d9bc9f7c"
Jul 17 20:42:51.260906 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.260886    2572 generic.go:356] "Generic (PLEG): container finished" podID="8412454b-ac97-4060-aa42-d8a246555c5a" containerID="506ce444d9cdf9dba4bb94245da239971d3fdebedb46143456a0dbc70cae302d" exitCode=6
Jul 17 20:42:51.260995 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.260935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" event={"ID":"8412454b-ac97-4060-aa42-d8a246555c5a","Type":"ContainerDied","Data":"506ce444d9cdf9dba4bb94245da239971d3fdebedb46143456a0dbc70cae302d"}
Jul 17 20:42:51.261167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.261152    2572 scope.go:117] "RemoveContainer" containerID="506ce444d9cdf9dba4bb94245da239971d3fdebedb46143456a0dbc70cae302d"
Jul 17 20:42:51.302840 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.302816    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"]
Jul 17 20:42:51.308260 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.308226    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-29rqt"]
Jul 17 20:42:51.853032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:51.853002    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" path="/var/lib/kubelet/pods/d664825b-4264-45ba-9b75-90be8e2fd108/volumes"
Jul 17 20:42:52.267545 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:52.267509    2572 generic.go:356] "Generic (PLEG): container finished" podID="8412454b-ac97-4060-aa42-d8a246555c5a" containerID="7f88b8f9e8728cce3c29913961c5e2d5adbbcdc562e68bc3104825fbde0fa8b6" exitCode=6
Jul 17 20:42:52.267982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:52.267593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" event={"ID":"8412454b-ac97-4060-aa42-d8a246555c5a","Type":"ContainerDied","Data":"7f88b8f9e8728cce3c29913961c5e2d5adbbcdc562e68bc3104825fbde0fa8b6"}
Jul 17 20:42:52.267982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:52.267655    2572 scope.go:117] "RemoveContainer" containerID="506ce444d9cdf9dba4bb94245da239971d3fdebedb46143456a0dbc70cae302d"
Jul 17 20:42:52.267982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:52.267861    2572 scope.go:117] "RemoveContainer" containerID="7f88b8f9e8728cce3c29913961c5e2d5adbbcdc562e68bc3104825fbde0fa8b6"
Jul 17 20:42:52.268150 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:52.268096    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr_odh-ai-gateway-infra(8412454b-ac97-4060-aa42-d8a246555c5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" podUID="8412454b-ac97-4060-aa42-d8a246555c5a"
Jul 17 20:42:53.245360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.245321    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:42:53.275359 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.275327    2572 scope.go:117] "RemoveContainer" containerID="7f88b8f9e8728cce3c29913961c5e2d5adbbcdc562e68bc3104825fbde0fa8b6"
Jul 17 20:42:53.275761 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:42:53.275556    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr_odh-ai-gateway-infra(8412454b-ac97-4060-aa42-d8a246555c5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" podUID="8412454b-ac97-4060-aa42-d8a246555c5a"
Jul 17 20:42:53.793735 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.793702    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"]
Jul 17 20:42:53.794031 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794019    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.794108 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794032    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.794108 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794060    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.794108 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794068    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.794108 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794080    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.794108 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794086    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.794382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794132    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.794382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794141    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a246af9d-542e-49b0-ba21-e164dbc5a5b1" containerName="revoke-keys"
Jul 17 20:42:53.794382 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.794150    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:42:53.798266 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.798244    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.804542 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.804523    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-13e7d9\"/\"openshift-service-ca.crt\""
Jul 17 20:42:53.805724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.805702    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-13e7d9\"/\"test-model-13e7d9-kserve-self-signed-certs\""
Jul 17 20:42:53.806595 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.806206    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-13e7d9\"/\"default-dockercfg-cqfsq\""
Jul 17 20:42:53.809580 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.806944    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-13e7d9\"/\"kube-root-ca.crt\""
Jul 17 20:42:53.815017 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.814990    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"]
Jul 17 20:42:53.845548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.845514    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-model-cache\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.845548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.845549    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ws5z\" (UniqueName: \"kubernetes.io/projected/b3ea12c1-f059-47c7-9770-365138bc4145-kube-api-access-9ws5z\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.845814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.845574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-home\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.845814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.845597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea12c1-f059-47c7-9770-365138bc4145-tls-certs\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.845814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.845643    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-tmp-dir\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.845814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.845732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-dshm\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947148 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea12c1-f059-47c7-9770-365138bc4145-tls-certs\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947148 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947152    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-tmp-dir\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947185    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-dshm\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-model-cache\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947298    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ws5z\" (UniqueName: \"kubernetes.io/projected/b3ea12c1-f059-47c7-9770-365138bc4145-kube-api-access-9ws5z\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947381    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-home\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947644 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947588    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-tmp-dir\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947732    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-model-cache\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.947815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.947796    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-home\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.949635 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.949615    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-dshm\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.949865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.949847    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea12c1-f059-47c7-9770-365138bc4145-tls-certs\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:53.963507 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:53.963478    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ws5z\" (UniqueName: \"kubernetes.io/projected/b3ea12c1-f059-47c7-9770-365138bc4145-kube-api-access-9ws5z\") pod \"test-model-13e7d9-kserve-86b78c89dc-ts4kc\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") " pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:54.019140 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:54.019111    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:42:54.112121 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:54.112015    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:42:54.253424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:54.253394    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"]
Jul 17 20:42:54.256277 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:42:54.256251    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ea12c1_f059_47c7_9770_365138bc4145.slice/crio-2defaf7a857d2ee4e52b55a991fe9e3722cfa97680934d2469c3042fe1621756 WatchSource:0}: Error finding container 2defaf7a857d2ee4e52b55a991fe9e3722cfa97680934d2469c3042fe1621756: Status 404 returned error can't find the container with id 2defaf7a857d2ee4e52b55a991fe9e3722cfa97680934d2469c3042fe1621756
Jul 17 20:42:54.280727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:54.280698    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc" event={"ID":"b3ea12c1-f059-47c7-9770-365138bc4145","Type":"ContainerStarted","Data":"2defaf7a857d2ee4e52b55a991fe9e3722cfa97680934d2469c3042fe1621756"}
Jul 17 20:42:56.292530 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:56.292495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc" event={"ID":"b3ea12c1-f059-47c7-9770-365138bc4145","Type":"ContainerStarted","Data":"2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5"}
Jul 17 20:42:56.314340 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:42:56.314281    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc" podStartSLOduration=2.165503009 podStartE2EDuration="3.314262315s" totalImagesPullingTime="1.148759306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:42:56.313290459 +0000 UTC m=+2811.007266619" watchObservedRunningTime="2026-07-17 20:42:56.314262315 +0000 UTC m=+2811.008238462"
Jul 17 20:43:04.112921 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:04.112884    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:43:04.113326 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:04.112941    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:43:04.125715 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:04.125686    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:43:04.341071 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:04.341031    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:43:04.490565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:04.490525    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:43:05.484174 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.484135    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"]
Jul 17 20:43:05.484678 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.484658    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d664825b-4264-45ba-9b75-90be8e2fd108" containerName="revoke-keys"
Jul 17 20:43:05.487743 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.487723    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"
Jul 17 20:43:05.514427 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.514400    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"]
Jul 17 20:43:05.548693 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.548662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7478d89-1371-4adc-8a9d-4a7415cee3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-khr7v\" (UID: \"b7478d89-1371-4adc-8a9d-4a7415cee3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"
Jul 17 20:43:05.649356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.649324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7478d89-1371-4adc-8a9d-4a7415cee3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-khr7v\" (UID: \"b7478d89-1371-4adc-8a9d-4a7415cee3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"
Jul 17 20:43:05.649955 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.649936    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7478d89-1371-4adc-8a9d-4a7415cee3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-khr7v\" (UID: \"b7478d89-1371-4adc-8a9d-4a7415cee3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"
Jul 17 20:43:05.797776 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.797697    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"
Jul 17 20:43:05.926975 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:05.926948    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"]
Jul 17 20:43:05.929793 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:43:05.929761    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7478d89_1371_4adc_8a9d_4a7415cee3fc.slice/crio-5409684c8aa4dec62a2df50578d96cb58b6492dc004c7e4566a6649c772a7800 WatchSource:0}: Error finding container 5409684c8aa4dec62a2df50578d96cb58b6492dc004c7e4566a6649c772a7800: Status 404 returned error can't find the container with id 5409684c8aa4dec62a2df50578d96cb58b6492dc004c7e4566a6649c772a7800
Jul 17 20:43:06.339909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:06.339874    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" event={"ID":"b7478d89-1371-4adc-8a9d-4a7415cee3fc","Type":"ContainerStarted","Data":"402b6d34f6d5da2eba81a79cf430d1941fd9a067d1151ef15dc497cccf8da04d"}
Jul 17 20:43:06.340100 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:06.339923    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" event={"ID":"b7478d89-1371-4adc-8a9d-4a7415cee3fc","Type":"ContainerStarted","Data":"5409684c8aa4dec62a2df50578d96cb58b6492dc004c7e4566a6649c772a7800"}
Jul 17 20:43:06.360964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:06.360912    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" podStartSLOduration=1.360894462 podStartE2EDuration="1.360894462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:43:06.359518503 +0000 UTC m=+2821.053494651" watchObservedRunningTime="2026-07-17 20:43:06.360894462 +0000 UTC m=+2821.054870629"
Jul 17 20:43:06.848483 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:06.848396    2572 scope.go:117] "RemoveContainer" containerID="7f88b8f9e8728cce3c29913961c5e2d5adbbcdc562e68bc3104825fbde0fa8b6"
Jul 17 20:43:07.345783 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:07.345750    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" event={"ID":"8412454b-ac97-4060-aa42-d8a246555c5a","Type":"ContainerStarted","Data":"dec4307580a299deb6b0e788bcbb131e9032f097a1559a7fa2f33eb93ac4c59c"}
Jul 17 20:43:07.347254 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:07.347230    2572 generic.go:356] "Generic (PLEG): container finished" podID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerID="402b6d34f6d5da2eba81a79cf430d1941fd9a067d1151ef15dc497cccf8da04d" exitCode=6
Jul 17 20:43:07.347364 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:07.347305    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" event={"ID":"b7478d89-1371-4adc-8a9d-4a7415cee3fc","Type":"ContainerDied","Data":"402b6d34f6d5da2eba81a79cf430d1941fd9a067d1151ef15dc497cccf8da04d"}
Jul 17 20:43:07.347451 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:07.347437    2572 scope.go:117] "RemoveContainer" containerID="402b6d34f6d5da2eba81a79cf430d1941fd9a067d1151ef15dc497cccf8da04d"
Jul 17 20:43:08.353660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.353564    2572 generic.go:356] "Generic (PLEG): container finished" podID="8412454b-ac97-4060-aa42-d8a246555c5a" containerID="dec4307580a299deb6b0e788bcbb131e9032f097a1559a7fa2f33eb93ac4c59c" exitCode=6
Jul 17 20:43:08.353660 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.353640    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" event={"ID":"8412454b-ac97-4060-aa42-d8a246555c5a","Type":"ContainerDied","Data":"dec4307580a299deb6b0e788bcbb131e9032f097a1559a7fa2f33eb93ac4c59c"}
Jul 17 20:43:08.354223 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.353706    2572 scope.go:117] "RemoveContainer" containerID="7f88b8f9e8728cce3c29913961c5e2d5adbbcdc562e68bc3104825fbde0fa8b6"
Jul 17 20:43:08.354223 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.353953    2572 scope.go:117] "RemoveContainer" containerID="dec4307580a299deb6b0e788bcbb131e9032f097a1559a7fa2f33eb93ac4c59c"
Jul 17 20:43:08.354223 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:08.354208    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr_odh-ai-gateway-infra(8412454b-ac97-4060-aa42-d8a246555c5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" podUID="8412454b-ac97-4060-aa42-d8a246555c5a"
Jul 17 20:43:08.355567 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.355547    2572 generic.go:356] "Generic (PLEG): container finished" podID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerID="73a28cff6911035030f0d6687b2d21064317b24c4a0072f386b5e754a80e2956" exitCode=6
Jul 17 20:43:08.355654 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.355600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" event={"ID":"b7478d89-1371-4adc-8a9d-4a7415cee3fc","Type":"ContainerDied","Data":"73a28cff6911035030f0d6687b2d21064317b24c4a0072f386b5e754a80e2956"}
Jul 17 20:43:08.355803 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.355790    2572 scope.go:117] "RemoveContainer" containerID="73a28cff6911035030f0d6687b2d21064317b24c4a0072f386b5e754a80e2956"
Jul 17 20:43:08.355944 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:08.355928    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-khr7v_odh-ai-gateway-infra(b7478d89-1371-4adc-8a9d-4a7415cee3fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc"
Jul 17 20:43:08.364293 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.364278    2572 scope.go:117] "RemoveContainer" containerID="402b6d34f6d5da2eba81a79cf430d1941fd9a067d1151ef15dc497cccf8da04d"
Jul 17 20:43:08.385712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:08.385653    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"]
Jul 17 20:43:09.363059 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:09.363031    2572 scope.go:117] "RemoveContainer" containerID="73a28cff6911035030f0d6687b2d21064317b24c4a0072f386b5e754a80e2956"
Jul 17 20:43:09.363365 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:09.363225    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-khr7v_odh-ai-gateway-infra(b7478d89-1371-4adc-8a9d-4a7415cee3fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc"
Jul 17 20:43:09.497248 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:09.497225    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"
Jul 17 20:43:09.581774 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:09.581742    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8412454b-ac97-4060-aa42-d8a246555c5a-maas-api-service-ca\") pod \"8412454b-ac97-4060-aa42-d8a246555c5a\" (UID: \"8412454b-ac97-4060-aa42-d8a246555c5a\") "
Jul 17 20:43:09.582117 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:09.582094    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8412454b-ac97-4060-aa42-d8a246555c5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8412454b-ac97-4060-aa42-d8a246555c5a" (UID: "8412454b-ac97-4060-aa42-d8a246555c5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:43:09.683035 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:09.682956    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8412454b-ac97-4060-aa42-d8a246555c5a-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:43:10.373595 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:10.373564    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"
Jul 17 20:43:10.374104 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:10.373595    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr" event={"ID":"8412454b-ac97-4060-aa42-d8a246555c5a","Type":"ContainerDied","Data":"ee9755111b4b1e93f998cb5af94001b9f97ca7dbd72509e226e303890f9d8176"}
Jul 17 20:43:10.374104 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:10.373670    2572 scope.go:117] "RemoveContainer" containerID="dec4307580a299deb6b0e788bcbb131e9032f097a1559a7fa2f33eb93ac4c59c"
Jul 17 20:43:10.399587 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:10.399561    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"]
Jul 17 20:43:10.410282 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:10.410258    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-lnzrr"]
Jul 17 20:43:11.855124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:11.855094    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" path="/var/lib/kubelet/pods/8412454b-ac97-4060-aa42-d8a246555c5a/volumes"
Jul 17 20:43:17.663289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663253    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"]
Jul 17 20:43:17.663718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663620    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:17.663718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663631    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:17.663718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663645    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:17.663718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663651    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:17.663718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663707    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:17.663718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663716    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:17.663718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.663722    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:17.666644 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.666620    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"
Jul 17 20:43:17.674960 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.674933    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"]
Jul 17 20:43:17.751421 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.751386    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a1db68-3ff6-402a-9c2b-2c957562d553-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rvdss\" (UID: \"52a1db68-3ff6-402a-9c2b-2c957562d553\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"
Jul 17 20:43:17.852269 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.852236    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a1db68-3ff6-402a-9c2b-2c957562d553-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rvdss\" (UID: \"52a1db68-3ff6-402a-9c2b-2c957562d553\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"
Jul 17 20:43:17.852819 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.852799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a1db68-3ff6-402a-9c2b-2c957562d553-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-rvdss\" (UID: \"52a1db68-3ff6-402a-9c2b-2c957562d553\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"
Jul 17 20:43:17.977500 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:17.977470    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"
Jul 17 20:43:18.107154 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:18.107093    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"]
Jul 17 20:43:18.108974 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:43:18.108947    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a1db68_3ff6_402a_9c2b_2c957562d553.slice/crio-b6856fed9a94094816a104ae595540547652629f3a75b82084fc7da1925a1575 WatchSource:0}: Error finding container b6856fed9a94094816a104ae595540547652629f3a75b82084fc7da1925a1575: Status 404 returned error can't find the container with id b6856fed9a94094816a104ae595540547652629f3a75b82084fc7da1925a1575
Jul 17 20:43:18.419641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:18.419604    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" event={"ID":"52a1db68-3ff6-402a-9c2b-2c957562d553","Type":"ContainerStarted","Data":"56541db4fcbf9c458ffcead750c18950ac1c54fb0b7e5ec6b1d031ee79eafea1"}
Jul 17 20:43:18.419807 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:18.419663    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" event={"ID":"52a1db68-3ff6-402a-9c2b-2c957562d553","Type":"ContainerStarted","Data":"b6856fed9a94094816a104ae595540547652629f3a75b82084fc7da1925a1575"}
Jul 17 20:43:18.437460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:18.437410    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" podStartSLOduration=1.43739554 podStartE2EDuration="1.43739554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:43:18.43670644 +0000 UTC m=+2833.130682589" watchObservedRunningTime="2026-07-17 20:43:18.43739554 +0000 UTC m=+2833.131371687"
Jul 17 20:43:19.425984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:19.425951    2572 generic.go:356] "Generic (PLEG): container finished" podID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerID="56541db4fcbf9c458ffcead750c18950ac1c54fb0b7e5ec6b1d031ee79eafea1" exitCode=6
Jul 17 20:43:19.425984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:19.425990    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" event={"ID":"52a1db68-3ff6-402a-9c2b-2c957562d553","Type":"ContainerDied","Data":"56541db4fcbf9c458ffcead750c18950ac1c54fb0b7e5ec6b1d031ee79eafea1"}
Jul 17 20:43:19.426520 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:19.426192    2572 scope.go:117] "RemoveContainer" containerID="56541db4fcbf9c458ffcead750c18950ac1c54fb0b7e5ec6b1d031ee79eafea1"
Jul 17 20:43:20.432472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:20.432375    2572 generic.go:356] "Generic (PLEG): container finished" podID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerID="f729805243ed26c715b2d7b9ebb01167e16152e0734f5905187f7b9ef4dd446b" exitCode=6
Jul 17 20:43:20.432861 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:20.432463    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" event={"ID":"52a1db68-3ff6-402a-9c2b-2c957562d553","Type":"ContainerDied","Data":"f729805243ed26c715b2d7b9ebb01167e16152e0734f5905187f7b9ef4dd446b"}
Jul 17 20:43:20.432861 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:20.432554    2572 scope.go:117] "RemoveContainer" containerID="56541db4fcbf9c458ffcead750c18950ac1c54fb0b7e5ec6b1d031ee79eafea1"
Jul 17 20:43:20.432861 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:20.432732    2572 scope.go:117] "RemoveContainer" containerID="f729805243ed26c715b2d7b9ebb01167e16152e0734f5905187f7b9ef4dd446b"
Jul 17 20:43:20.433011 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:20.432938    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rvdss_odh-ai-gateway-infra(52a1db68-3ff6-402a-9c2b-2c957562d553)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553"
Jul 17 20:43:21.439874 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:21.439841    2572 scope.go:117] "RemoveContainer" containerID="f729805243ed26c715b2d7b9ebb01167e16152e0734f5905187f7b9ef4dd446b"
Jul 17 20:43:21.440306 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:21.440020    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rvdss_odh-ai-gateway-infra(52a1db68-3ff6-402a-9c2b-2c957562d553)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553"
Jul 17 20:43:21.848200 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:21.848170    2572 scope.go:117] "RemoveContainer" containerID="73a28cff6911035030f0d6687b2d21064317b24c4a0072f386b5e754a80e2956"
Jul 17 20:43:22.446842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:22.446817    2572 generic.go:356] "Generic (PLEG): container finished" podID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerID="b1b2895b222ffaa9ef5dfcd5f125ab7c13ba201b4e46206474890cc9d6e3d2d1" exitCode=6
Jul 17 20:43:22.447158 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:22.446872    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" event={"ID":"b7478d89-1371-4adc-8a9d-4a7415cee3fc","Type":"ContainerDied","Data":"b1b2895b222ffaa9ef5dfcd5f125ab7c13ba201b4e46206474890cc9d6e3d2d1"}
Jul 17 20:43:22.447158 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:22.446916    2572 scope.go:117] "RemoveContainer" containerID="73a28cff6911035030f0d6687b2d21064317b24c4a0072f386b5e754a80e2956"
Jul 17 20:43:22.447768 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:22.447745    2572 scope.go:117] "RemoveContainer" containerID="b1b2895b222ffaa9ef5dfcd5f125ab7c13ba201b4e46206474890cc9d6e3d2d1"
Jul 17 20:43:22.448159 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:22.448134    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-khr7v_odh-ai-gateway-infra(b7478d89-1371-4adc-8a9d-4a7415cee3fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc"
Jul 17 20:43:23.473983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:23.473945    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"]
Jul 17 20:43:23.610090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:23.610063    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"
Jul 17 20:43:23.705806 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:23.705770    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7478d89-1371-4adc-8a9d-4a7415cee3fc-maas-api-service-ca\") pod \"b7478d89-1371-4adc-8a9d-4a7415cee3fc\" (UID: \"b7478d89-1371-4adc-8a9d-4a7415cee3fc\") "
Jul 17 20:43:23.706188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:23.706165    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7478d89-1371-4adc-8a9d-4a7415cee3fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7478d89-1371-4adc-8a9d-4a7415cee3fc" (UID: "b7478d89-1371-4adc-8a9d-4a7415cee3fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:43:23.807019 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:23.806931    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7478d89-1371-4adc-8a9d-4a7415cee3fc-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:43:24.457945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:24.457915    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"
Jul 17 20:43:24.458212 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:24.457915    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v" event={"ID":"b7478d89-1371-4adc-8a9d-4a7415cee3fc","Type":"ContainerDied","Data":"5409684c8aa4dec62a2df50578d96cb58b6492dc004c7e4566a6649c772a7800"}
Jul 17 20:43:24.458212 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:24.458071    2572 scope.go:117] "RemoveContainer" containerID="b1b2895b222ffaa9ef5dfcd5f125ab7c13ba201b4e46206474890cc9d6e3d2d1"
Jul 17 20:43:24.475029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:24.475002    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"]
Jul 17 20:43:24.478930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:24.478906    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-khr7v"]
Jul 17 20:43:25.854871 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:25.854840    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" path="/var/lib/kubelet/pods/b7478d89-1371-4adc-8a9d-4a7415cee3fc/volumes"
Jul 17 20:43:32.848799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:32.848771    2572 scope.go:117] "RemoveContainer" containerID="f729805243ed26c715b2d7b9ebb01167e16152e0734f5905187f7b9ef4dd446b"
Jul 17 20:43:32.992165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:32.992134    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:43:33.500328 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:33.500295    2572 generic.go:356] "Generic (PLEG): container finished" podID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerID="68dd081ef8464c0c8fbd6c86bd3e419fd71d896174219ee51dc501c39de05f42" exitCode=6
Jul 17 20:43:33.500514 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:33.500340    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" event={"ID":"52a1db68-3ff6-402a-9c2b-2c957562d553","Type":"ContainerDied","Data":"68dd081ef8464c0c8fbd6c86bd3e419fd71d896174219ee51dc501c39de05f42"}
Jul 17 20:43:33.500514 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:33.500390    2572 scope.go:117] "RemoveContainer" containerID="f729805243ed26c715b2d7b9ebb01167e16152e0734f5905187f7b9ef4dd446b"
Jul 17 20:43:33.500740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:33.500723    2572 scope.go:117] "RemoveContainer" containerID="68dd081ef8464c0c8fbd6c86bd3e419fd71d896174219ee51dc501c39de05f42"
Jul 17 20:43:33.500977 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:33.500949    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-rvdss_odh-ai-gateway-infra(52a1db68-3ff6-402a-9c2b-2c957562d553)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553"
Jul 17 20:43:34.531308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:34.531268    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"]
Jul 17 20:43:34.666523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:34.666501    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"
Jul 17 20:43:34.800337 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:34.800222    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a1db68-3ff6-402a-9c2b-2c957562d553-maas-api-service-ca\") pod \"52a1db68-3ff6-402a-9c2b-2c957562d553\" (UID: \"52a1db68-3ff6-402a-9c2b-2c957562d553\") "
Jul 17 20:43:34.800608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:34.800585    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a1db68-3ff6-402a-9c2b-2c957562d553-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52a1db68-3ff6-402a-9c2b-2c957562d553" (UID: "52a1db68-3ff6-402a-9c2b-2c957562d553"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:43:34.901188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:34.901150    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a1db68-3ff6-402a-9c2b-2c957562d553-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:43:35.512118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:35.512086    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"
Jul 17 20:43:35.512118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:35.512110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss" event={"ID":"52a1db68-3ff6-402a-9c2b-2c957562d553","Type":"ContainerDied","Data":"b6856fed9a94094816a104ae595540547652629f3a75b82084fc7da1925a1575"}
Jul 17 20:43:35.512341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:35.512179    2572 scope.go:117] "RemoveContainer" containerID="68dd081ef8464c0c8fbd6c86bd3e419fd71d896174219ee51dc501c39de05f42"
Jul 17 20:43:35.555063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:35.555021    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"]
Jul 17 20:43:35.556589 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:35.556565    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-rvdss"]
Jul 17 20:43:35.854207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:35.854127    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" path="/var/lib/kubelet/pods/52a1db68-3ff6-402a-9c2b-2c957562d553/volumes"
Jul 17 20:43:45.816011 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:45.815975    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"]
Jul 17 20:43:47.006416 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006381    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"]
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006725    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006737    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8412454b-ac97-4060-aa42-d8a246555c5a" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006745    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006751    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006764    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006769    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006780    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006785    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006793    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006799    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006807    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.006811 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006811    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.007206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006858    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:47.007206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006866    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:47.007206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006872    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.007206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006879    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.007206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.006886    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a1db68-3ff6-402a-9c2b-2c957562d553" containerName="revoke-keys"
Jul 17 20:43:47.009777 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.009754    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"
Jul 17 20:43:47.012854 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.012831    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:43:47.019570 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.019548    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"]
Jul 17 20:43:47.091424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.091388    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b4aabc-fa84-42fa-8810-b4c91bcd6675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv\" (UID: \"35b4aabc-fa84-42fa-8810-b4c91bcd6675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"
Jul 17 20:43:47.192059 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.192014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b4aabc-fa84-42fa-8810-b4c91bcd6675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv\" (UID: \"35b4aabc-fa84-42fa-8810-b4c91bcd6675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"
Jul 17 20:43:47.192679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.192659    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b4aabc-fa84-42fa-8810-b4c91bcd6675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv\" (UID: \"35b4aabc-fa84-42fa-8810-b4c91bcd6675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"
Jul 17 20:43:47.320428 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.320333    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"
Jul 17 20:43:47.459127 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.459092    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"]
Jul 17 20:43:47.464183 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:43:47.464155    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b4aabc_fa84_42fa_8810_b4c91bcd6675.slice/crio-16f8e611f1a697ff925cddb51eefc8d0545a273ff4702d413c02bdca5f00cb25 WatchSource:0}: Error finding container 16f8e611f1a697ff925cddb51eefc8d0545a273ff4702d413c02bdca5f00cb25: Status 404 returned error can't find the container with id 16f8e611f1a697ff925cddb51eefc8d0545a273ff4702d413c02bdca5f00cb25
Jul 17 20:43:47.570992 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:47.570910    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv" event={"ID":"35b4aabc-fa84-42fa-8810-b4c91bcd6675","Type":"ContainerStarted","Data":"16f8e611f1a697ff925cddb51eefc8d0545a273ff4702d413c02bdca5f00cb25"}
Jul 17 20:43:48.577197 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:48.577161    2572 generic.go:356] "Generic (PLEG): container finished" podID="35b4aabc-fa84-42fa-8810-b4c91bcd6675" containerID="387e6c1fb916ce52e0d42d8234969ee4419bcfc6407370e5338e3b752a4780d6" exitCode=0
Jul 17 20:43:48.577589 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:48.577227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv" event={"ID":"35b4aabc-fa84-42fa-8810-b4c91bcd6675","Type":"ContainerDied","Data":"387e6c1fb916ce52e0d42d8234969ee4419bcfc6407370e5338e3b752a4780d6"}
Jul 17 20:43:49.531034 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.531001    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"]
Jul 17 20:43:49.531374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.531361    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:49.531421 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.531377    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:49.531456 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.531433    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7478d89-1371-4adc-8a9d-4a7415cee3fc" containerName="revoke-keys"
Jul 17 20:43:49.534374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.534357    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"
Jul 17 20:43:49.544757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.544732    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"]
Jul 17 20:43:49.611939 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.611899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd42371-cf00-44a8-a21d-593b18c90b6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm\" (UID: \"8bd42371-cf00-44a8-a21d-593b18c90b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"
Jul 17 20:43:49.706063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.706030    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"
Jul 17 20:43:49.712833 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.712813    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd42371-cf00-44a8-a21d-593b18c90b6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm\" (UID: \"8bd42371-cf00-44a8-a21d-593b18c90b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"
Jul 17 20:43:49.713399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.713382    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd42371-cf00-44a8-a21d-593b18c90b6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm\" (UID: \"8bd42371-cf00-44a8-a21d-593b18c90b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"
Jul 17 20:43:49.813627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.813531    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b4aabc-fa84-42fa-8810-b4c91bcd6675-maas-api-service-ca\") pod \"35b4aabc-fa84-42fa-8810-b4c91bcd6675\" (UID: \"35b4aabc-fa84-42fa-8810-b4c91bcd6675\") "
Jul 17 20:43:49.813916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.813893    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b4aabc-fa84-42fa-8810-b4c91bcd6675-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35b4aabc-fa84-42fa-8810-b4c91bcd6675" (UID: "35b4aabc-fa84-42fa-8810-b4c91bcd6675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:43:49.844359 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.844331    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"
Jul 17 20:43:49.914420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.914387    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b4aabc-fa84-42fa-8810-b4c91bcd6675-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:43:49.974118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:49.974038    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"]
Jul 17 20:43:49.976113 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:43:49.976087    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd42371_cf00_44a8_a21d_593b18c90b6e.slice/crio-a0aefb08612624fe754212614f5032090f0942ae30adb72e8972eaa365d15961 WatchSource:0}: Error finding container a0aefb08612624fe754212614f5032090f0942ae30adb72e8972eaa365d15961: Status 404 returned error can't find the container with id a0aefb08612624fe754212614f5032090f0942ae30adb72e8972eaa365d15961
Jul 17 20:43:50.587772 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:50.587731    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" event={"ID":"8bd42371-cf00-44a8-a21d-593b18c90b6e","Type":"ContainerStarted","Data":"ce5ebb228442144f111c887df1675c42d7e67e3c3b2dfb6ec7c7e55bb37eaa91"}
Jul 17 20:43:50.587961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:50.587827    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" event={"ID":"8bd42371-cf00-44a8-a21d-593b18c90b6e","Type":"ContainerStarted","Data":"a0aefb08612624fe754212614f5032090f0942ae30adb72e8972eaa365d15961"}
Jul 17 20:43:50.588906 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:50.588881    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv"
Jul 17 20:43:50.589070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:50.588881    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv" event={"ID":"35b4aabc-fa84-42fa-8810-b4c91bcd6675","Type":"ContainerDied","Data":"16f8e611f1a697ff925cddb51eefc8d0545a273ff4702d413c02bdca5f00cb25"}
Jul 17 20:43:50.589070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:50.589000    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f8e611f1a697ff925cddb51eefc8d0545a273ff4702d413c02bdca5f00cb25"
Jul 17 20:43:50.604372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:50.604323    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" podStartSLOduration=1.604310043 podStartE2EDuration="1.604310043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:43:50.602244173 +0000 UTC m=+2865.296220320" watchObservedRunningTime="2026-07-17 20:43:50.604310043 +0000 UTC m=+2865.298286188"
Jul 17 20:43:51.595352 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:51.595315    2572 generic.go:356] "Generic (PLEG): container finished" podID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerID="ce5ebb228442144f111c887df1675c42d7e67e3c3b2dfb6ec7c7e55bb37eaa91" exitCode=6
Jul 17 20:43:51.595732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:51.595372    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" event={"ID":"8bd42371-cf00-44a8-a21d-593b18c90b6e","Type":"ContainerDied","Data":"ce5ebb228442144f111c887df1675c42d7e67e3c3b2dfb6ec7c7e55bb37eaa91"}
Jul 17 20:43:51.595732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:51.595572    2572 scope.go:117] "RemoveContainer" containerID="ce5ebb228442144f111c887df1675c42d7e67e3c3b2dfb6ec7c7e55bb37eaa91"
Jul 17 20:43:51.890923 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:51.890844    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:43:52.603757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:52.603657    2572 generic.go:356] "Generic (PLEG): container finished" podID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerID="4001f2461df6d12740eecde5046de08e4644b0c6c052f1e19b94c84889086d53" exitCode=6
Jul 17 20:43:52.603757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:52.603713    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" event={"ID":"8bd42371-cf00-44a8-a21d-593b18c90b6e","Type":"ContainerDied","Data":"4001f2461df6d12740eecde5046de08e4644b0c6c052f1e19b94c84889086d53"}
Jul 17 20:43:52.604301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:52.603805    2572 scope.go:117] "RemoveContainer" containerID="ce5ebb228442144f111c887df1675c42d7e67e3c3b2dfb6ec7c7e55bb37eaa91"
Jul 17 20:43:52.604301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:52.603997    2572 scope.go:117] "RemoveContainer" containerID="4001f2461df6d12740eecde5046de08e4644b0c6c052f1e19b94c84889086d53"
Jul 17 20:43:52.604301 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:52.604222    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm_odh-ai-gateway-infra(8bd42371-cf00-44a8-a21d-593b18c90b6e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e"
Jul 17 20:43:53.609776 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:53.609743    2572 scope.go:117] "RemoveContainer" containerID="4001f2461df6d12740eecde5046de08e4644b0c6c052f1e19b94c84889086d53"
Jul 17 20:43:53.610264 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:53.609975    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm_odh-ai-gateway-infra(8bd42371-cf00-44a8-a21d-593b18c90b6e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e"
Jul 17 20:43:55.521041 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.521004    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"]
Jul 17 20:43:55.521462 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.521356    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b4aabc-fa84-42fa-8810-b4c91bcd6675" containerName="revoke-keys"
Jul 17 20:43:55.521462 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.521370    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b4aabc-fa84-42fa-8810-b4c91bcd6675" containerName="revoke-keys"
Jul 17 20:43:55.521462 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.521436    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b4aabc-fa84-42fa-8810-b4c91bcd6675" containerName="revoke-keys"
Jul 17 20:43:55.525732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.525712    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"
Jul 17 20:43:55.543164 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.543130    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"]
Jul 17 20:43:55.562261 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.562224    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9ff26-ff23-478a-a65a-502e534add9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-m42kq\" (UID: \"aba9ff26-ff23-478a-a65a-502e534add9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"
Jul 17 20:43:55.663340 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.663296    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9ff26-ff23-478a-a65a-502e534add9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-m42kq\" (UID: \"aba9ff26-ff23-478a-a65a-502e534add9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"
Jul 17 20:43:55.663916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.663896    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9ff26-ff23-478a-a65a-502e534add9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-m42kq\" (UID: \"aba9ff26-ff23-478a-a65a-502e534add9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"
Jul 17 20:43:55.836181 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.836091    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"
Jul 17 20:43:55.968576 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:55.968552    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"]
Jul 17 20:43:55.970680 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:43:55.970652    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba9ff26_ff23_478a_a65a_502e534add9f.slice/crio-143c157a1490cc35255b161344bacf5ae1718110b44c8e84016d8f9308519029 WatchSource:0}: Error finding container 143c157a1490cc35255b161344bacf5ae1718110b44c8e84016d8f9308519029: Status 404 returned error can't find the container with id 143c157a1490cc35255b161344bacf5ae1718110b44c8e84016d8f9308519029
Jul 17 20:43:56.624436 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:56.624402    2572 generic.go:356] "Generic (PLEG): container finished" podID="aba9ff26-ff23-478a-a65a-502e534add9f" containerID="cc8a5d30fe64834eded41199854d3316edf7a397f5f788fb30e4ca0231139692" exitCode=6
Jul 17 20:43:56.624878 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:56.624464    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" event={"ID":"aba9ff26-ff23-478a-a65a-502e534add9f","Type":"ContainerDied","Data":"cc8a5d30fe64834eded41199854d3316edf7a397f5f788fb30e4ca0231139692"}
Jul 17 20:43:56.624878 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:56.624577    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" event={"ID":"aba9ff26-ff23-478a-a65a-502e534add9f","Type":"ContainerStarted","Data":"143c157a1490cc35255b161344bacf5ae1718110b44c8e84016d8f9308519029"}
Jul 17 20:43:56.624878 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:56.624812    2572 scope.go:117] "RemoveContainer" containerID="cc8a5d30fe64834eded41199854d3316edf7a397f5f788fb30e4ca0231139692"
Jul 17 20:43:57.631728 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:57.631691    2572 generic.go:356] "Generic (PLEG): container finished" podID="aba9ff26-ff23-478a-a65a-502e534add9f" containerID="167d9ad845a2ec3d8ac18fcd9cb0a7075de7f46e264b7ad5e14870d75d72d9bf" exitCode=6
Jul 17 20:43:57.632191 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:57.631781    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" event={"ID":"aba9ff26-ff23-478a-a65a-502e534add9f","Type":"ContainerDied","Data":"167d9ad845a2ec3d8ac18fcd9cb0a7075de7f46e264b7ad5e14870d75d72d9bf"}
Jul 17 20:43:57.632191 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:57.631839    2572 scope.go:117] "RemoveContainer" containerID="cc8a5d30fe64834eded41199854d3316edf7a397f5f788fb30e4ca0231139692"
Jul 17 20:43:57.632191 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:57.632031    2572 scope.go:117] "RemoveContainer" containerID="167d9ad845a2ec3d8ac18fcd9cb0a7075de7f46e264b7ad5e14870d75d72d9bf"
Jul 17 20:43:57.632351 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:57.632283    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-m42kq_odh-ai-gateway-infra(aba9ff26-ff23-478a-a65a-502e534add9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" podUID="aba9ff26-ff23-478a-a65a-502e534add9f"
Jul 17 20:43:58.638128 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:43:58.638099    2572 scope.go:117] "RemoveContainer" containerID="167d9ad845a2ec3d8ac18fcd9cb0a7075de7f46e264b7ad5e14870d75d72d9bf"
Jul 17 20:43:58.638488 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:43:58.638259    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-m42kq_odh-ai-gateway-infra(aba9ff26-ff23-478a-a65a-502e534add9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" podUID="aba9ff26-ff23-478a-a65a-502e534add9f"
Jul 17 20:44:00.836203 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:00.836165    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc" podUID="b3ea12c1-f059-47c7-9770-365138bc4145" containerName="main" containerID="cri-o://2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5" gracePeriod=45
Jul 17 20:44:01.084433 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.084407    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:44:01.214120 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214093    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-tmp-dir\") pod \"b3ea12c1-f059-47c7-9770-365138bc4145\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") "
Jul 17 20:44:01.214289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214149    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea12c1-f059-47c7-9770-365138bc4145-tls-certs\") pod \"b3ea12c1-f059-47c7-9770-365138bc4145\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") "
Jul 17 20:44:01.214289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214185    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-dshm\") pod \"b3ea12c1-f059-47c7-9770-365138bc4145\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") "
Jul 17 20:44:01.214289 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214231    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ws5z\" (UniqueName: \"kubernetes.io/projected/b3ea12c1-f059-47c7-9770-365138bc4145-kube-api-access-9ws5z\") pod \"b3ea12c1-f059-47c7-9770-365138bc4145\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") "
Jul 17 20:44:01.214465 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214307    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-model-cache\") pod \"b3ea12c1-f059-47c7-9770-365138bc4145\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") "
Jul 17 20:44:01.214465 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214331    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-home\") pod \"b3ea12c1-f059-47c7-9770-365138bc4145\" (UID: \"b3ea12c1-f059-47c7-9770-365138bc4145\") "
Jul 17 20:44:01.214465 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214417    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3ea12c1-f059-47c7-9770-365138bc4145" (UID: "b3ea12c1-f059-47c7-9770-365138bc4145"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:44:01.214616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214569    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-tmp-dir\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:01.214616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214576    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3ea12c1-f059-47c7-9770-365138bc4145" (UID: "b3ea12c1-f059-47c7-9770-365138bc4145"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:44:01.214690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.214632    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-home" (OuterVolumeSpecName: "home") pod "b3ea12c1-f059-47c7-9770-365138bc4145" (UID: "b3ea12c1-f059-47c7-9770-365138bc4145"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:44:01.216498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.216471    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ea12c1-f059-47c7-9770-365138bc4145-kube-api-access-9ws5z" (OuterVolumeSpecName: "kube-api-access-9ws5z") pod "b3ea12c1-f059-47c7-9770-365138bc4145" (UID: "b3ea12c1-f059-47c7-9770-365138bc4145"). InnerVolumeSpecName "kube-api-access-9ws5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:44:01.216498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.216488    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-dshm" (OuterVolumeSpecName: "dshm") pod "b3ea12c1-f059-47c7-9770-365138bc4145" (UID: "b3ea12c1-f059-47c7-9770-365138bc4145"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:44:01.216645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.216499    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ea12c1-f059-47c7-9770-365138bc4145-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3ea12c1-f059-47c7-9770-365138bc4145" (UID: "b3ea12c1-f059-47c7-9770-365138bc4145"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 20:44:01.315616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.315582    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-model-cache\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:01.315616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.315608    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-home\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:01.315616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.315616    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea12c1-f059-47c7-9770-365138bc4145-tls-certs\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:01.315616 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.315625    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ea12c1-f059-47c7-9770-365138bc4145-dshm\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:01.315880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.315633    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ws5z\" (UniqueName: \"kubernetes.io/projected/b3ea12c1-f059-47c7-9770-365138bc4145-kube-api-access-9ws5z\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:01.652681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.652592    2572 generic.go:356] "Generic (PLEG): container finished" podID="b3ea12c1-f059-47c7-9770-365138bc4145" containerID="2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5" exitCode=0
Jul 17 20:44:01.652681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.652638    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"
Jul 17 20:44:01.652877 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.652677    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc" event={"ID":"b3ea12c1-f059-47c7-9770-365138bc4145","Type":"ContainerDied","Data":"2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5"}
Jul 17 20:44:01.652877 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.652717    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc" event={"ID":"b3ea12c1-f059-47c7-9770-365138bc4145","Type":"ContainerDied","Data":"2defaf7a857d2ee4e52b55a991fe9e3722cfa97680934d2469c3042fe1621756"}
Jul 17 20:44:01.652877 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.652736    2572 scope.go:117] "RemoveContainer" containerID="2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5"
Jul 17 20:44:01.661669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.661652    2572 scope.go:117] "RemoveContainer" containerID="2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5"
Jul 17 20:44:01.661932 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:01.661911    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5\": container with ID starting with 2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5 not found: ID does not exist" containerID="2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5"
Jul 17 20:44:01.661979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.661940    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5"} err="failed to get container status \"2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5\": rpc error: code = NotFound desc = could not find container \"2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5\": container with ID starting with 2c0f70b78719b7b9914db4ed5875834e35af626be6d3eee43e7001870add4af5 not found: ID does not exist"
Jul 17 20:44:01.675962 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.675936    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"]
Jul 17 20:44:01.680614 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.680593    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-13e7d9/test-model-13e7d9-kserve-86b78c89dc-ts4kc"]
Jul 17 20:44:01.859676 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:01.859638    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ea12c1-f059-47c7-9770-365138bc4145" path="/var/lib/kubelet/pods/b3ea12c1-f059-47c7-9770-365138bc4145/volumes"
Jul 17 20:44:03.387151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:03.387117    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:44:06.848310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:06.848275    2572 scope.go:117] "RemoveContainer" containerID="4001f2461df6d12740eecde5046de08e4644b0c6c052f1e19b94c84889086d53"
Jul 17 20:44:07.682816 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.682779    2572 generic.go:356] "Generic (PLEG): container finished" podID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerID="eb3aade0e297b5941286293f03c40993a684efa2f504a1ccb0cfffad3773ab46" exitCode=6
Jul 17 20:44:07.683009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.682825    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" event={"ID":"8bd42371-cf00-44a8-a21d-593b18c90b6e","Type":"ContainerDied","Data":"eb3aade0e297b5941286293f03c40993a684efa2f504a1ccb0cfffad3773ab46"}
Jul 17 20:44:07.683009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.682913    2572 scope.go:117] "RemoveContainer" containerID="4001f2461df6d12740eecde5046de08e4644b0c6c052f1e19b94c84889086d53"
Jul 17 20:44:07.683209 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.683188    2572 scope.go:117] "RemoveContainer" containerID="eb3aade0e297b5941286293f03c40993a684efa2f504a1ccb0cfffad3773ab46"
Jul 17 20:44:07.683412 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:07.683393    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm_odh-ai-gateway-infra(8bd42371-cf00-44a8-a21d-593b18c90b6e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e"
Jul 17 20:44:07.727882 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.727845    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"]
Jul 17 20:44:07.728254 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.728238    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ea12c1-f059-47c7-9770-365138bc4145" containerName="main"
Jul 17 20:44:07.728325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.728256    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ea12c1-f059-47c7-9770-365138bc4145" containerName="main"
Jul 17 20:44:07.728325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.728325    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ea12c1-f059-47c7-9770-365138bc4145" containerName="main"
Jul 17 20:44:07.732479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.732461    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"
Jul 17 20:44:07.738655 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.738627    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"]
Jul 17 20:44:07.868666 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.868637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8b1073-9c78-4f29-a52c-b8d037f93f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-x4htt\" (UID: \"fd8b1073-9c78-4f29-a52c-b8d037f93f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"
Jul 17 20:44:07.969842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.969806    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8b1073-9c78-4f29-a52c-b8d037f93f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-x4htt\" (UID: \"fd8b1073-9c78-4f29-a52c-b8d037f93f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"
Jul 17 20:44:07.970435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:07.970416    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8b1073-9c78-4f29-a52c-b8d037f93f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-x4htt\" (UID: \"fd8b1073-9c78-4f29-a52c-b8d037f93f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"
Jul 17 20:44:08.044828 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.044793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"
Jul 17 20:44:08.173453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.173364    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"]
Jul 17 20:44:08.176033 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:44:08.176008    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8b1073_9c78_4f29_a52c_b8d037f93f9a.slice/crio-fb6a4b09caf6ca1d2f3157c3c36fbc7e0e08b53ac2784aa46a2db1cf27174308 WatchSource:0}: Error finding container fb6a4b09caf6ca1d2f3157c3c36fbc7e0e08b53ac2784aa46a2db1cf27174308: Status 404 returned error can't find the container with id fb6a4b09caf6ca1d2f3157c3c36fbc7e0e08b53ac2784aa46a2db1cf27174308
Jul 17 20:44:08.688573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.688467    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" event={"ID":"fd8b1073-9c78-4f29-a52c-b8d037f93f9a","Type":"ContainerStarted","Data":"6efb29360df85b9b3148341861d9537bddd2e5bcc48886ead22df5ba43ce70eb"}
Jul 17 20:44:08.688573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.688525    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" event={"ID":"fd8b1073-9c78-4f29-a52c-b8d037f93f9a","Type":"ContainerStarted","Data":"fb6a4b09caf6ca1d2f3157c3c36fbc7e0e08b53ac2784aa46a2db1cf27174308"}
Jul 17 20:44:08.707991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.707937    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" podStartSLOduration=1.7079200669999999 podStartE2EDuration="1.707920067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:44:08.706810583 +0000 UTC m=+2883.400786730" watchObservedRunningTime="2026-07-17 20:44:08.707920067 +0000 UTC m=+2883.401896213"
Jul 17 20:44:08.712719 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.712692    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"]
Jul 17 20:44:08.840898 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.840877    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"
Jul 17 20:44:08.979302 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.979264    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd42371-cf00-44a8-a21d-593b18c90b6e-maas-api-service-ca\") pod \"8bd42371-cf00-44a8-a21d-593b18c90b6e\" (UID: \"8bd42371-cf00-44a8-a21d-593b18c90b6e\") "
Jul 17 20:44:08.979749 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:08.979676    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd42371-cf00-44a8-a21d-593b18c90b6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8bd42371-cf00-44a8-a21d-593b18c90b6e" (UID: "8bd42371-cf00-44a8-a21d-593b18c90b6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:44:09.080714 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.080678    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd42371-cf00-44a8-a21d-593b18c90b6e-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:09.695691 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.695664    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"
Jul 17 20:44:09.695891 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.695701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm" event={"ID":"8bd42371-cf00-44a8-a21d-593b18c90b6e","Type":"ContainerDied","Data":"a0aefb08612624fe754212614f5032090f0942ae30adb72e8972eaa365d15961"}
Jul 17 20:44:09.695891 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.695768    2572 scope.go:117] "RemoveContainer" containerID="eb3aade0e297b5941286293f03c40993a684efa2f504a1ccb0cfffad3773ab46"
Jul 17 20:44:09.697198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.697171    2572 generic.go:356] "Generic (PLEG): container finished" podID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerID="6efb29360df85b9b3148341861d9537bddd2e5bcc48886ead22df5ba43ce70eb" exitCode=6
Jul 17 20:44:09.697319 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.697222    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" event={"ID":"fd8b1073-9c78-4f29-a52c-b8d037f93f9a","Type":"ContainerDied","Data":"6efb29360df85b9b3148341861d9537bddd2e5bcc48886ead22df5ba43ce70eb"}
Jul 17 20:44:09.697426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.697413    2572 scope.go:117] "RemoveContainer" containerID="6efb29360df85b9b3148341861d9537bddd2e5bcc48886ead22df5ba43ce70eb"
Jul 17 20:44:09.744410 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.744384    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"]
Jul 17 20:44:09.752291 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.752268    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4vxlm"]
Jul 17 20:44:09.852905 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:09.852872    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" path="/var/lib/kubelet/pods/8bd42371-cf00-44a8-a21d-593b18c90b6e/volumes"
Jul 17 20:44:10.704453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:10.704364    2572 generic.go:356] "Generic (PLEG): container finished" podID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerID="ac59d175075e1e9f3ceef9a73c34ac15cd0f769fcf49724b2aa74c0c9bedd25a" exitCode=6
Jul 17 20:44:10.704859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:10.704445    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" event={"ID":"fd8b1073-9c78-4f29-a52c-b8d037f93f9a","Type":"ContainerDied","Data":"ac59d175075e1e9f3ceef9a73c34ac15cd0f769fcf49724b2aa74c0c9bedd25a"}
Jul 17 20:44:10.704859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:10.704508    2572 scope.go:117] "RemoveContainer" containerID="6efb29360df85b9b3148341861d9537bddd2e5bcc48886ead22df5ba43ce70eb"
Jul 17 20:44:10.704859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:10.704672    2572 scope.go:117] "RemoveContainer" containerID="ac59d175075e1e9f3ceef9a73c34ac15cd0f769fcf49724b2aa74c0c9bedd25a"
Jul 17 20:44:10.705030 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:10.704879    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-x4htt_odh-ai-gateway-infra(fd8b1073-9c78-4f29-a52c-b8d037f93f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a"
Jul 17 20:44:11.711367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:11.711339    2572 scope.go:117] "RemoveContainer" containerID="ac59d175075e1e9f3ceef9a73c34ac15cd0f769fcf49724b2aa74c0c9bedd25a"
Jul 17 20:44:11.711749 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:11.711517    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-x4htt_odh-ai-gateway-infra(fd8b1073-9c78-4f29-a52c-b8d037f93f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a"
Jul 17 20:44:12.848235 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:12.848205    2572 scope.go:117] "RemoveContainer" containerID="167d9ad845a2ec3d8ac18fcd9cb0a7075de7f46e264b7ad5e14870d75d72d9bf"
Jul 17 20:44:13.135139 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135111    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"]
Jul 17 20:44:13.135445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135430    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:13.135513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135448    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:13.135513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135460    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:13.135513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135468    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:13.135600 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135572    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:13.135600 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135585    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:13.135600 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.135595    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:13.138457 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.138438    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"
Jul 17 20:44:13.146591 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.146564    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"]
Jul 17 20:44:13.218376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.218343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72f75b7-d1ee-4a0f-90d3-1d872af07c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc\" (UID: \"d72f75b7-d1ee-4a0f-90d3-1d872af07c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"
Jul 17 20:44:13.318799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.318758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72f75b7-d1ee-4a0f-90d3-1d872af07c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc\" (UID: \"d72f75b7-d1ee-4a0f-90d3-1d872af07c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"
Jul 17 20:44:13.319399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.319379    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72f75b7-d1ee-4a0f-90d3-1d872af07c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc\" (UID: \"d72f75b7-d1ee-4a0f-90d3-1d872af07c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"
Jul 17 20:44:13.450349 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.450317    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"
Jul 17 20:44:13.587833 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.587802    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"]
Jul 17 20:44:13.589650 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:44:13.589611    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72f75b7_d1ee_4a0f_90d3_1d872af07c72.slice/crio-228cee634c4340a897de9319b79d6a8c79be8da1b52d3bfe6a5cb108fc73f6b4 WatchSource:0}: Error finding container 228cee634c4340a897de9319b79d6a8c79be8da1b52d3bfe6a5cb108fc73f6b4: Status 404 returned error can't find the container with id 228cee634c4340a897de9319b79d6a8c79be8da1b52d3bfe6a5cb108fc73f6b4
Jul 17 20:44:13.721926 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.721897    2572 generic.go:356] "Generic (PLEG): container finished" podID="aba9ff26-ff23-478a-a65a-502e534add9f" containerID="bda45a2408957e00fd90fb11c03c7fe0488b32d446dddf862085e3762481333a" exitCode=6
Jul 17 20:44:13.722126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.721968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" event={"ID":"aba9ff26-ff23-478a-a65a-502e534add9f","Type":"ContainerDied","Data":"bda45a2408957e00fd90fb11c03c7fe0488b32d446dddf862085e3762481333a"}
Jul 17 20:44:13.722126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.722022    2572 scope.go:117] "RemoveContainer" containerID="167d9ad845a2ec3d8ac18fcd9cb0a7075de7f46e264b7ad5e14870d75d72d9bf"
Jul 17 20:44:13.722325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.722305    2572 scope.go:117] "RemoveContainer" containerID="bda45a2408957e00fd90fb11c03c7fe0488b32d446dddf862085e3762481333a"
Jul 17 20:44:13.722537 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:13.722511    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-m42kq_odh-ai-gateway-infra(aba9ff26-ff23-478a-a65a-502e534add9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" podUID="aba9ff26-ff23-478a-a65a-502e534add9f"
Jul 17 20:44:13.723371 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:13.723338    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc" event={"ID":"d72f75b7-d1ee-4a0f-90d3-1d872af07c72","Type":"ContainerStarted","Data":"228cee634c4340a897de9319b79d6a8c79be8da1b52d3bfe6a5cb108fc73f6b4"}
Jul 17 20:44:14.730746 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:14.730712    2572 generic.go:356] "Generic (PLEG): container finished" podID="d72f75b7-d1ee-4a0f-90d3-1d872af07c72" containerID="c3d0e7c6a9478f43236cf98006c79563ad0a2afb668262e80e0e30fd868f4fea" exitCode=0
Jul 17 20:44:14.731190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:14.730796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc" event={"ID":"d72f75b7-d1ee-4a0f-90d3-1d872af07c72","Type":"ContainerDied","Data":"c3d0e7c6a9478f43236cf98006c79563ad0a2afb668262e80e0e30fd868f4fea"}
Jul 17 20:44:14.753409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:14.753379    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"]
Jul 17 20:44:14.882458 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:14.882435    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"
Jul 17 20:44:15.032699 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.032613    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9ff26-ff23-478a-a65a-502e534add9f-maas-api-service-ca\") pod \"aba9ff26-ff23-478a-a65a-502e534add9f\" (UID: \"aba9ff26-ff23-478a-a65a-502e534add9f\") "
Jul 17 20:44:15.033000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.032970    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba9ff26-ff23-478a-a65a-502e534add9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aba9ff26-ff23-478a-a65a-502e534add9f" (UID: "aba9ff26-ff23-478a-a65a-502e534add9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:44:15.133305 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.133265    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9ff26-ff23-478a-a65a-502e534add9f-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:15.738459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.738364    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"
Jul 17 20:44:15.738459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.738396    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq" event={"ID":"aba9ff26-ff23-478a-a65a-502e534add9f","Type":"ContainerDied","Data":"143c157a1490cc35255b161344bacf5ae1718110b44c8e84016d8f9308519029"}
Jul 17 20:44:15.738992 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.738469    2572 scope.go:117] "RemoveContainer" containerID="bda45a2408957e00fd90fb11c03c7fe0488b32d446dddf862085e3762481333a"
Jul 17 20:44:15.766753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.766723    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"]
Jul 17 20:44:15.768183 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.768164    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-m42kq"]
Jul 17 20:44:15.854481 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.854449    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" path="/var/lib/kubelet/pods/aba9ff26-ff23-478a-a65a-502e534add9f/volumes"
Jul 17 20:44:15.872963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.872942    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"
Jul 17 20:44:15.940126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.940095    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72f75b7-d1ee-4a0f-90d3-1d872af07c72-maas-api-service-ca\") pod \"d72f75b7-d1ee-4a0f-90d3-1d872af07c72\" (UID: \"d72f75b7-d1ee-4a0f-90d3-1d872af07c72\") "
Jul 17 20:44:15.940445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:15.940424    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72f75b7-d1ee-4a0f-90d3-1d872af07c72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d72f75b7-d1ee-4a0f-90d3-1d872af07c72" (UID: "d72f75b7-d1ee-4a0f-90d3-1d872af07c72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:44:16.041573 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:16.041483    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72f75b7-d1ee-4a0f-90d3-1d872af07c72-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:16.744856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:16.744822    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc"
Jul 17 20:44:16.744856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:16.744838    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc" event={"ID":"d72f75b7-d1ee-4a0f-90d3-1d872af07c72","Type":"ContainerDied","Data":"228cee634c4340a897de9319b79d6a8c79be8da1b52d3bfe6a5cb108fc73f6b4"}
Jul 17 20:44:16.745482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:16.744928    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228cee634c4340a897de9319b79d6a8c79be8da1b52d3bfe6a5cb108fc73f6b4"
Jul 17 20:44:21.498317 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498278    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"]
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498598    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498612    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd42371-cf00-44a8-a21d-593b18c90b6e" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498632    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72f75b7-d1ee-4a0f-90d3-1d872af07c72" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498637    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72f75b7-d1ee-4a0f-90d3-1d872af07c72" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498646    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498651    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498659    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498664    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498670    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.498708 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498675    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.499007 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498725    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.499007 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498733    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72f75b7-d1ee-4a0f-90d3-1d872af07c72" containerName="revoke-keys"
Jul 17 20:44:21.499007 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.498738    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:21.501843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.501827    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"
Jul 17 20:44:21.511130 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.511104    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"]
Jul 17 20:44:21.592063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.592018    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b7760-1332-4b33-8f9e-90f194bcf07d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-xtbch\" (UID: \"cc7b7760-1332-4b33-8f9e-90f194bcf07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"
Jul 17 20:44:21.693428 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.693386    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b7760-1332-4b33-8f9e-90f194bcf07d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-xtbch\" (UID: \"cc7b7760-1332-4b33-8f9e-90f194bcf07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"
Jul 17 20:44:21.694035 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.694014    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b7760-1332-4b33-8f9e-90f194bcf07d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-xtbch\" (UID: \"cc7b7760-1332-4b33-8f9e-90f194bcf07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"
Jul 17 20:44:21.813298 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.813204    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"
Jul 17 20:44:21.946009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:21.945968    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"]
Jul 17 20:44:21.948017 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:44:21.947990    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7b7760_1332_4b33_8f9e_90f194bcf07d.slice/crio-423850c9b123c6090fb4f63fd574a3d499f2a45572a9e25ac5cd9b1b8ade58a1 WatchSource:0}: Error finding container 423850c9b123c6090fb4f63fd574a3d499f2a45572a9e25ac5cd9b1b8ade58a1: Status 404 returned error can't find the container with id 423850c9b123c6090fb4f63fd574a3d499f2a45572a9e25ac5cd9b1b8ade58a1
Jul 17 20:44:22.773153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:22.773115    2572 generic.go:356] "Generic (PLEG): container finished" podID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerID="7b4282715b74a88cbdd9afe1c5a4c2b83f2f4a8aa745b879b8d8c1ca78d72358" exitCode=6
Jul 17 20:44:22.773540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:22.773187    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" event={"ID":"cc7b7760-1332-4b33-8f9e-90f194bcf07d","Type":"ContainerDied","Data":"7b4282715b74a88cbdd9afe1c5a4c2b83f2f4a8aa745b879b8d8c1ca78d72358"}
Jul 17 20:44:22.773540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:22.773240    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" event={"ID":"cc7b7760-1332-4b33-8f9e-90f194bcf07d","Type":"ContainerStarted","Data":"423850c9b123c6090fb4f63fd574a3d499f2a45572a9e25ac5cd9b1b8ade58a1"}
Jul 17 20:44:22.773540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:22.773339    2572 scope.go:117] "RemoveContainer" containerID="7b4282715b74a88cbdd9afe1c5a4c2b83f2f4a8aa745b879b8d8c1ca78d72358"
Jul 17 20:44:22.848427 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:22.848398    2572 scope.go:117] "RemoveContainer" containerID="ac59d175075e1e9f3ceef9a73c34ac15cd0f769fcf49724b2aa74c0c9bedd25a"
Jul 17 20:44:23.692126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.691380    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"]
Jul 17 20:44:23.692126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.691625    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98" podUID="c15470ed-8a04-4819-a609-0cf6f6472739" containerName="maas-api" containerID="cri-o://a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76" gracePeriod=30
Jul 17 20:44:23.779826 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.779791    2572 generic.go:356] "Generic (PLEG): container finished" podID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerID="ff1b79f57bec8a0d896c0e2f6251367727e6fe087031b4d99f8e02a76d9600fe" exitCode=6
Jul 17 20:44:23.780309 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.779872    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" event={"ID":"fd8b1073-9c78-4f29-a52c-b8d037f93f9a","Type":"ContainerDied","Data":"ff1b79f57bec8a0d896c0e2f6251367727e6fe087031b4d99f8e02a76d9600fe"}
Jul 17 20:44:23.780309 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.779953    2572 scope.go:117] "RemoveContainer" containerID="ac59d175075e1e9f3ceef9a73c34ac15cd0f769fcf49724b2aa74c0c9bedd25a"
Jul 17 20:44:23.780309 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.780272    2572 scope.go:117] "RemoveContainer" containerID="ff1b79f57bec8a0d896c0e2f6251367727e6fe087031b4d99f8e02a76d9600fe"
Jul 17 20:44:23.780509 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:23.780488    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-x4htt_odh-ai-gateway-infra(fd8b1073-9c78-4f29-a52c-b8d037f93f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a"
Jul 17 20:44:23.781862 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.781844    2572 generic.go:356] "Generic (PLEG): container finished" podID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerID="f38905078befbbf38d06b1e7f1773e5252ebd495da6c525a9ea2e8fcd7bae286" exitCode=6
Jul 17 20:44:23.781941 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.781925    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" event={"ID":"cc7b7760-1332-4b33-8f9e-90f194bcf07d","Type":"ContainerDied","Data":"f38905078befbbf38d06b1e7f1773e5252ebd495da6c525a9ea2e8fcd7bae286"}
Jul 17 20:44:23.782111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.782095    2572 scope.go:117] "RemoveContainer" containerID="f38905078befbbf38d06b1e7f1773e5252ebd495da6c525a9ea2e8fcd7bae286"
Jul 17 20:44:23.782259 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:23.782242    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-xtbch_odh-ai-gateway-infra(cc7b7760-1332-4b33-8f9e-90f194bcf07d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d"
Jul 17 20:44:23.806576 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.806553    2572 scope.go:117] "RemoveContainer" containerID="7b4282715b74a88cbdd9afe1c5a4c2b83f2f4a8aa745b879b8d8c1ca78d72358"
Jul 17 20:44:23.944995 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:23.944932    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:44:24.116993 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.116958    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls\") pod \"c15470ed-8a04-4819-a609-0cf6f6472739\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") "
Jul 17 20:44:24.117221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.117041    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b844\" (UniqueName: \"kubernetes.io/projected/c15470ed-8a04-4819-a609-0cf6f6472739-kube-api-access-6b844\") pod \"c15470ed-8a04-4819-a609-0cf6f6472739\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") "
Jul 17 20:44:24.117221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.117102    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15470ed-8a04-4819-a609-0cf6f6472739-openshift-service-ca\") pod \"c15470ed-8a04-4819-a609-0cf6f6472739\" (UID: \"c15470ed-8a04-4819-a609-0cf6f6472739\") "
Jul 17 20:44:24.117430 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.117396    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15470ed-8a04-4819-a609-0cf6f6472739-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c15470ed-8a04-4819-a609-0cf6f6472739" (UID: "c15470ed-8a04-4819-a609-0cf6f6472739"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:44:24.119197 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.119171    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c15470ed-8a04-4819-a609-0cf6f6472739" (UID: "c15470ed-8a04-4819-a609-0cf6f6472739"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 20:44:24.119301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.119222    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15470ed-8a04-4819-a609-0cf6f6472739-kube-api-access-6b844" (OuterVolumeSpecName: "kube-api-access-6b844") pod "c15470ed-8a04-4819-a609-0cf6f6472739" (UID: "c15470ed-8a04-4819-a609-0cf6f6472739"). InnerVolumeSpecName "kube-api-access-6b844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:44:24.217854 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.217826    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b844\" (UniqueName: \"kubernetes.io/projected/c15470ed-8a04-4819-a609-0cf6f6472739-kube-api-access-6b844\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:24.217854 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.217853    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15470ed-8a04-4819-a609-0cf6f6472739-openshift-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:24.218039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.217863    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15470ed-8a04-4819-a609-0cf6f6472739-maas-api-tls\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:24.789557 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.789524    2572 generic.go:356] "Generic (PLEG): container finished" podID="c15470ed-8a04-4819-a609-0cf6f6472739" containerID="a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76" exitCode=0
Jul 17 20:44:24.789999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.789565    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"
Jul 17 20:44:24.789999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.789597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98" event={"ID":"c15470ed-8a04-4819-a609-0cf6f6472739","Type":"ContainerDied","Data":"a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76"}
Jul 17 20:44:24.789999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.789643    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98" event={"ID":"c15470ed-8a04-4819-a609-0cf6f6472739","Type":"ContainerDied","Data":"9427e4909204ed412ca9d89eca7701801812238ac1ff877e7532810e04a3b58b"}
Jul 17 20:44:24.789999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.789665    2572 scope.go:117] "RemoveContainer" containerID="a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76"
Jul 17 20:44:24.791369 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.791354    2572 scope.go:117] "RemoveContainer" containerID="f38905078befbbf38d06b1e7f1773e5252ebd495da6c525a9ea2e8fcd7bae286"
Jul 17 20:44:24.791548 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:24.791531    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-xtbch_odh-ai-gateway-infra(cc7b7760-1332-4b33-8f9e-90f194bcf07d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d"
Jul 17 20:44:24.798245 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.798221    2572 scope.go:117] "RemoveContainer" containerID="a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76"
Jul 17 20:44:24.798518 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:24.798496    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76\": container with ID starting with a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76 not found: ID does not exist" containerID="a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76"
Jul 17 20:44:24.798584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.798528    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76"} err="failed to get container status \"a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76\": rpc error: code = NotFound desc = could not find container \"a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76\": container with ID starting with a178a8be446984964534da413a25e3e7c5811691c028df5df1945e01ed828c76 not found: ID does not exist"
Jul 17 20:44:24.810898 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.810866    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"]
Jul 17 20:44:24.850954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.850923    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"]
Jul 17 20:44:24.856518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.856491    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f0f4f-f6b988c54-g8d98"]
Jul 17 20:44:24.943126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:24.943104    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"
Jul 17 20:44:25.023709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.023668    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8b1073-9c78-4f29-a52c-b8d037f93f9a-maas-api-service-ca\") pod \"fd8b1073-9c78-4f29-a52c-b8d037f93f9a\" (UID: \"fd8b1073-9c78-4f29-a52c-b8d037f93f9a\") "
Jul 17 20:44:25.024154 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.024127    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8b1073-9c78-4f29-a52c-b8d037f93f9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd8b1073-9c78-4f29-a52c-b8d037f93f9a" (UID: "fd8b1073-9c78-4f29-a52c-b8d037f93f9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:44:25.024304 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.024232    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8b1073-9c78-4f29-a52c-b8d037f93f9a-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:25.797331 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.797296    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt" event={"ID":"fd8b1073-9c78-4f29-a52c-b8d037f93f9a","Type":"ContainerDied","Data":"fb6a4b09caf6ca1d2f3157c3c36fbc7e0e08b53ac2784aa46a2db1cf27174308"}
Jul 17 20:44:25.797803 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.797333    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"
Jul 17 20:44:25.797803 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.797356    2572 scope.go:117] "RemoveContainer" containerID="ff1b79f57bec8a0d896c0e2f6251367727e6fe087031b4d99f8e02a76d9600fe"
Jul 17 20:44:25.818718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.818688    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"]
Jul 17 20:44:25.829936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.829908    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-x4htt"]
Jul 17 20:44:25.853283 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.853261    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15470ed-8a04-4819-a609-0cf6f6472739" path="/var/lib/kubelet/pods/c15470ed-8a04-4819-a609-0cf6f6472739/volumes"
Jul 17 20:44:25.853582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:25.853571    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" path="/var/lib/kubelet/pods/fd8b1073-9c78-4f29-a52c-b8d037f93f9a/volumes"
Jul 17 20:44:26.880039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:26.880005    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:44:36.847960 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:36.847931    2572 scope.go:117] "RemoveContainer" containerID="f38905078befbbf38d06b1e7f1773e5252ebd495da6c525a9ea2e8fcd7bae286"
Jul 17 20:44:37.852700 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:37.852667    2572 generic.go:356] "Generic (PLEG): container finished" podID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerID="d3a44cff6ab9997a5eed1a060807ab5c7387b9c28597b29291d17d95912dfc68" exitCode=6
Jul 17 20:44:37.853134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:37.852720    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" event={"ID":"cc7b7760-1332-4b33-8f9e-90f194bcf07d","Type":"ContainerDied","Data":"d3a44cff6ab9997a5eed1a060807ab5c7387b9c28597b29291d17d95912dfc68"}
Jul 17 20:44:37.853134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:37.852764    2572 scope.go:117] "RemoveContainer" containerID="f38905078befbbf38d06b1e7f1773e5252ebd495da6c525a9ea2e8fcd7bae286"
Jul 17 20:44:37.853134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:37.853025    2572 scope.go:117] "RemoveContainer" containerID="d3a44cff6ab9997a5eed1a060807ab5c7387b9c28597b29291d17d95912dfc68"
Jul 17 20:44:37.853265 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:37.853234    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-xtbch_odh-ai-gateway-infra(cc7b7760-1332-4b33-8f9e-90f194bcf07d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d"
Jul 17 20:44:38.879103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:38.879070    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"]
Jul 17 20:44:39.021348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.021326    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"
Jul 17 20:44:39.133733 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.133650    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b7760-1332-4b33-8f9e-90f194bcf07d-maas-api-service-ca\") pod \"cc7b7760-1332-4b33-8f9e-90f194bcf07d\" (UID: \"cc7b7760-1332-4b33-8f9e-90f194bcf07d\") "
Jul 17 20:44:39.134001 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.133976    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7b7760-1332-4b33-8f9e-90f194bcf07d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc7b7760-1332-4b33-8f9e-90f194bcf07d" (UID: "cc7b7760-1332-4b33-8f9e-90f194bcf07d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:44:39.234969 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.234934    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b7760-1332-4b33-8f9e-90f194bcf07d-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:44:39.864718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.864688    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch" event={"ID":"cc7b7760-1332-4b33-8f9e-90f194bcf07d","Type":"ContainerDied","Data":"423850c9b123c6090fb4f63fd574a3d499f2a45572a9e25ac5cd9b1b8ade58a1"}
Jul 17 20:44:39.864718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.864711    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"
Jul 17 20:44:39.864946 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.864753    2572 scope.go:117] "RemoveContainer" containerID="d3a44cff6ab9997a5eed1a060807ab5c7387b9c28597b29291d17d95912dfc68"
Jul 17 20:44:39.881690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.881662    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"]
Jul 17 20:44:39.888533 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:39.888493    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-xtbch"]
Jul 17 20:44:41.853732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:41.853693    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" path="/var/lib/kubelet/pods/cc7b7760-1332-4b33-8f9e-90f194bcf07d/volumes"
Jul 17 20:44:42.349451 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349421    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"]
Jul 17 20:44:42.349798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349781    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349802    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349817    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15470ed-8a04-4819-a609-0cf6f6472739" containerName="maas-api"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349825    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15470ed-8a04-4819-a609-0cf6f6472739" containerName="maas-api"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349839    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349847    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349857    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349865    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349875    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349883    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.349896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349898    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349906    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349922    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.349930    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.350028    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.350040    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.350070    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.350078    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba9ff26-ff23-478a-a65a-502e534add9f" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.350087    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15470ed-8a04-4819-a609-0cf6f6472739" containerName="maas-api"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.350096    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.350409 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.350105    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7b7760-1332-4b33-8f9e-90f194bcf07d" containerName="revoke-keys"
Jul 17 20:44:42.354621 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.354597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.357033 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.357011    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 20:44:42.357255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.357234    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:44:42.357896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.357880    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3a4159-serving-cert\""
Jul 17 20:44:42.357972 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.357882    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mxjrb\""
Jul 17 20:44:42.362607 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.362583    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"]
Jul 17 20:44:42.365907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.365881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf87g\" (UniqueName: \"kubernetes.io/projected/ae872c5f-099e-4665-895e-76d0fae17e42-kube-api-access-nf87g\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.366022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.365924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae872c5f-099e-4665-895e-76d0fae17e42-maas-api-tls\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.366022 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.365962    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae872c5f-099e-4665-895e-76d0fae17e42-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.467018 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.466987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf87g\" (UniqueName: \"kubernetes.io/projected/ae872c5f-099e-4665-895e-76d0fae17e42-kube-api-access-nf87g\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.467018 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.467031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae872c5f-099e-4665-895e-76d0fae17e42-maas-api-tls\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.467281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.467107    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae872c5f-099e-4665-895e-76d0fae17e42-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.467701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.467679    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae872c5f-099e-4665-895e-76d0fae17e42-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.469632 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.469610    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae872c5f-099e-4665-895e-76d0fae17e42-maas-api-tls\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.477148 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.477127    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf87g\" (UniqueName: \"kubernetes.io/projected/ae872c5f-099e-4665-895e-76d0fae17e42-kube-api-access-nf87g\") pod \"maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.667149 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.667042    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:42.800610 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.800586    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"]
Jul 17 20:44:42.802585 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:44:42.802559    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae872c5f_099e_4665_895e_76d0fae17e42.slice/crio-8b35321726bc769dcabc0886015963c48f8ecb95c5f3d68a6fe17a751b2fe3bf WatchSource:0}: Error finding container 8b35321726bc769dcabc0886015963c48f8ecb95c5f3d68a6fe17a751b2fe3bf: Status 404 returned error can't find the container with id 8b35321726bc769dcabc0886015963c48f8ecb95c5f3d68a6fe17a751b2fe3bf
Jul 17 20:44:42.880547 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:42.880513    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5" event={"ID":"ae872c5f-099e-4665-895e-76d0fae17e42","Type":"ContainerStarted","Data":"8b35321726bc769dcabc0886015963c48f8ecb95c5f3d68a6fe17a751b2fe3bf"}
Jul 17 20:44:43.886508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:43.886421    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5" event={"ID":"ae872c5f-099e-4665-895e-76d0fae17e42","Type":"ContainerStarted","Data":"04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65"}
Jul 17 20:44:43.886967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:43.886524    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:43.907559 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:43.907507    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5" podStartSLOduration=1.177750915 podStartE2EDuration="1.907489185s" totalImagesPullingTime="729.73827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:44:43.906229778 +0000 UTC m=+2918.600205926" watchObservedRunningTime="2026-07-17 20:44:43.907489185 +0000 UTC m=+2918.601465362"
Jul 17 20:44:47.797123 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:47.797072    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:44:49.580366 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.580332    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"]
Jul 17 20:44:49.580847 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.580691    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8b1073-9c78-4f29-a52c-b8d037f93f9a" containerName="revoke-keys"
Jul 17 20:44:49.583765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.583747    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"
Jul 17 20:44:49.586027 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.586007    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:44:49.593135 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.593100    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"]
Jul 17 20:44:49.626751 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.626710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69055cdc-8732-4eb9-aad0-077f61c76d0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf\" (UID: \"69055cdc-8732-4eb9-aad0-077f61c76d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"
Jul 17 20:44:49.727116 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.727075    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69055cdc-8732-4eb9-aad0-077f61c76d0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf\" (UID: \"69055cdc-8732-4eb9-aad0-077f61c76d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"
Jul 17 20:44:49.727690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.727672    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69055cdc-8732-4eb9-aad0-077f61c76d0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf\" (UID: \"69055cdc-8732-4eb9-aad0-077f61c76d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"
Jul 17 20:44:49.894564 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.894463    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"
Jul 17 20:44:49.896079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:49.896038    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:44:50.029585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.029555    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"]
Jul 17 20:44:50.355534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.355499    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"]
Jul 17 20:44:50.360070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.360035    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.362175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.362157    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ad0e32-serving-cert\""
Jul 17 20:44:50.373543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.373518    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"]
Jul 17 20:44:50.433206 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.433180    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f573bda4-3bda-405c-8c59-c6923f064c99-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.433320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.433232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.433320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.433296    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsrr\" (UniqueName: \"kubernetes.io/projected/f573bda4-3bda-405c-8c59-c6923f064c99-kube-api-access-7zsrr\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.534420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.534342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsrr\" (UniqueName: \"kubernetes.io/projected/f573bda4-3bda-405c-8c59-c6923f064c99-kube-api-access-7zsrr\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.534420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.534409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f573bda4-3bda-405c-8c59-c6923f064c99-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.534617 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.534455    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.534617 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:50.534575    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-serving-cert: secret "maas-api-e2e-api-b-ad0e32-serving-cert" not found
Jul 17 20:44:50.534695 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:50.534642    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls podName:f573bda4-3bda-405c-8c59-c6923f064c99 nodeName:}" failed. No retries permitted until 2026-07-17 20:44:51.034620835 +0000 UTC m=+2925.728596963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls") pod "maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw" (UID: "f573bda4-3bda-405c-8c59-c6923f064c99") : secret "maas-api-e2e-api-b-ad0e32-serving-cert" not found
Jul 17 20:44:50.535118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.535099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f573bda4-3bda-405c-8c59-c6923f064c99-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.545718 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.545688    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsrr\" (UniqueName: \"kubernetes.io/projected/f573bda4-3bda-405c-8c59-c6923f064c99-kube-api-access-7zsrr\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:50.918466 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.918428    2572 generic.go:356] "Generic (PLEG): container finished" podID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerID="02e1b2bf3c1185a8c73912fbfb00218657c4d49463f14fd2f6e97ff5270805d0" exitCode=6
Jul 17 20:44:50.918824 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.918479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" event={"ID":"69055cdc-8732-4eb9-aad0-077f61c76d0f","Type":"ContainerDied","Data":"02e1b2bf3c1185a8c73912fbfb00218657c4d49463f14fd2f6e97ff5270805d0"}
Jul 17 20:44:50.918824 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.918529    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" event={"ID":"69055cdc-8732-4eb9-aad0-077f61c76d0f","Type":"ContainerStarted","Data":"4bf356f2f43c24e1d61166af338c333c2bbb42009790bc2c17307207bc782c5e"}
Jul 17 20:44:50.918824 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:50.918731    2572 scope.go:117] "RemoveContainer" containerID="02e1b2bf3c1185a8c73912fbfb00218657c4d49463f14fd2f6e97ff5270805d0"
Jul 17 20:44:51.039720 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.039688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:51.042252 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.042222    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls\") pod \"maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:51.269931 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.269901    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:51.400952 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.400881    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"]
Jul 17 20:44:51.403013 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:44:51.402990    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf573bda4_3bda_405c_8c59_c6923f064c99.slice/crio-042c1d82c1844d09cbec9aefa207982213bb5ccd65b11a15eeaad29f0e3bf237 WatchSource:0}: Error finding container 042c1d82c1844d09cbec9aefa207982213bb5ccd65b11a15eeaad29f0e3bf237: Status 404 returned error can't find the container with id 042c1d82c1844d09cbec9aefa207982213bb5ccd65b11a15eeaad29f0e3bf237
Jul 17 20:44:51.924506 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.924474    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw" event={"ID":"f573bda4-3bda-405c-8c59-c6923f064c99","Type":"ContainerStarted","Data":"6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a"}
Jul 17 20:44:51.924894 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.924626    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:51.924894 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.924651    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw" event={"ID":"f573bda4-3bda-405c-8c59-c6923f064c99","Type":"ContainerStarted","Data":"042c1d82c1844d09cbec9aefa207982213bb5ccd65b11a15eeaad29f0e3bf237"}
Jul 17 20:44:51.927064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.927028    2572 generic.go:356] "Generic (PLEG): container finished" podID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerID="4c7d7154929393bde20cad288e4041dbd252aac44e6a6d3a4f843b2a7baf8416" exitCode=6
Jul 17 20:44:51.927169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.927088    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" event={"ID":"69055cdc-8732-4eb9-aad0-077f61c76d0f","Type":"ContainerDied","Data":"4c7d7154929393bde20cad288e4041dbd252aac44e6a6d3a4f843b2a7baf8416"}
Jul 17 20:44:51.927169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.927154    2572 scope.go:117] "RemoveContainer" containerID="02e1b2bf3c1185a8c73912fbfb00218657c4d49463f14fd2f6e97ff5270805d0"
Jul 17 20:44:51.927316 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.927293    2572 scope.go:117] "RemoveContainer" containerID="4c7d7154929393bde20cad288e4041dbd252aac44e6a6d3a4f843b2a7baf8416"
Jul 17 20:44:51.927511 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:51.927489    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf_odh-ai-gateway-infra(69055cdc-8732-4eb9-aad0-077f61c76d0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f"
Jul 17 20:44:51.943410 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:51.943362    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw" podStartSLOduration=1.494320307 podStartE2EDuration="1.943345948s" totalImagesPullingTime="449.025641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:44:51.941642964 +0000 UTC m=+2926.635619122" watchObservedRunningTime="2026-07-17 20:44:51.943345948 +0000 UTC m=+2926.637322093"
Jul 17 20:44:52.934201 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:52.934169    2572 scope.go:117] "RemoveContainer" containerID="4c7d7154929393bde20cad288e4041dbd252aac44e6a6d3a4f843b2a7baf8416"
Jul 17 20:44:52.934692 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:52.934418    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf_odh-ai-gateway-infra(69055cdc-8732-4eb9-aad0-077f61c76d0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f"
Jul 17 20:44:55.566195 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:55.566159    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"]
Jul 17 20:44:55.570688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:55.570671    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"
Jul 17 20:44:55.576408 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:55.576380    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"]
Jul 17 20:44:55.679842 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:55.679807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661abf74-7e3e-48d2-8088-45da1b382787-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8\" (UID: \"661abf74-7e3e-48d2-8088-45da1b382787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"
Jul 17 20:44:55.780402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:55.780360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661abf74-7e3e-48d2-8088-45da1b382787-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8\" (UID: \"661abf74-7e3e-48d2-8088-45da1b382787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"
Jul 17 20:44:55.781003 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:55.780981    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661abf74-7e3e-48d2-8088-45da1b382787-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8\" (UID: \"661abf74-7e3e-48d2-8088-45da1b382787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"
Jul 17 20:44:55.881441 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:55.881366    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"
Jul 17 20:44:56.018123 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:56.018100    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"]
Jul 17 20:44:56.020135 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:44:56.020108    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661abf74_7e3e_48d2_8088_45da1b382787.slice/crio-ff6f98c382d7a3164e81ef0d2bdf5ff5050c37fcfb28e5b6ec1a76272072dca0 WatchSource:0}: Error finding container ff6f98c382d7a3164e81ef0d2bdf5ff5050c37fcfb28e5b6ec1a76272072dca0: Status 404 returned error can't find the container with id ff6f98c382d7a3164e81ef0d2bdf5ff5050c37fcfb28e5b6ec1a76272072dca0
Jul 17 20:44:56.958548 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:56.958450    2572 generic.go:356] "Generic (PLEG): container finished" podID="661abf74-7e3e-48d2-8088-45da1b382787" containerID="a3921e528471cbb77f9f4f04768fdfefe34e62323a96f3098fbd45114afd8f6d" exitCode=6
Jul 17 20:44:56.958915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:56.958534    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" event={"ID":"661abf74-7e3e-48d2-8088-45da1b382787","Type":"ContainerDied","Data":"a3921e528471cbb77f9f4f04768fdfefe34e62323a96f3098fbd45114afd8f6d"}
Jul 17 20:44:56.958915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:56.958597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" event={"ID":"661abf74-7e3e-48d2-8088-45da1b382787","Type":"ContainerStarted","Data":"ff6f98c382d7a3164e81ef0d2bdf5ff5050c37fcfb28e5b6ec1a76272072dca0"}
Jul 17 20:44:56.958915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:56.958740    2572 scope.go:117] "RemoveContainer" containerID="a3921e528471cbb77f9f4f04768fdfefe34e62323a96f3098fbd45114afd8f6d"
Jul 17 20:44:57.788793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.788763    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"]
Jul 17 20:44:57.792105 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.792088    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"
Jul 17 20:44:57.799399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.799236    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"]
Jul 17 20:44:57.896609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.896524    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f7e34-c6ae-4ce4-a38c-ce681cff1ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb\" (UID: \"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"
Jul 17 20:44:57.939407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.939379    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:44:57.973377 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.973341    2572 generic.go:356] "Generic (PLEG): container finished" podID="661abf74-7e3e-48d2-8088-45da1b382787" containerID="c897e68a3c65ed0c9989e860a7069f0e164ea131bbe3eab531bf83e8d0136fe9" exitCode=6
Jul 17 20:44:57.973798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.973423    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" event={"ID":"661abf74-7e3e-48d2-8088-45da1b382787","Type":"ContainerDied","Data":"c897e68a3c65ed0c9989e860a7069f0e164ea131bbe3eab531bf83e8d0136fe9"}
Jul 17 20:44:57.973798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.973533    2572 scope.go:117] "RemoveContainer" containerID="a3921e528471cbb77f9f4f04768fdfefe34e62323a96f3098fbd45114afd8f6d"
Jul 17 20:44:57.973798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.973720    2572 scope.go:117] "RemoveContainer" containerID="c897e68a3c65ed0c9989e860a7069f0e164ea131bbe3eab531bf83e8d0136fe9"
Jul 17 20:44:57.973991 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:57.973944    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8_odh-ai-gateway-infra(661abf74-7e3e-48d2-8088-45da1b382787)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" podUID="661abf74-7e3e-48d2-8088-45da1b382787"
Jul 17 20:44:57.997813 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.997782    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f7e34-c6ae-4ce4-a38c-ce681cff1ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb\" (UID: \"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"
Jul 17 20:44:57.998422 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:57.998401    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f7e34-c6ae-4ce4-a38c-ce681cff1ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb\" (UID: \"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"
Jul 17 20:44:58.103927 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:58.103890    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"
Jul 17 20:44:58.233675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:58.233647    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"]
Jul 17 20:44:58.236230 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:44:58.236189    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533f7e34_c6ae_4ce4_a38c_ce681cff1ef5.slice/crio-5d57ad685ff26f40dbc261f950bce9ad24f7526a21d74bcd05bcd1d222d4734a WatchSource:0}: Error finding container 5d57ad685ff26f40dbc261f950bce9ad24f7526a21d74bcd05bcd1d222d4734a: Status 404 returned error can't find the container with id 5d57ad685ff26f40dbc261f950bce9ad24f7526a21d74bcd05bcd1d222d4734a
Jul 17 20:44:58.980129 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:58.980100    2572 scope.go:117] "RemoveContainer" containerID="c897e68a3c65ed0c9989e860a7069f0e164ea131bbe3eab531bf83e8d0136fe9"
Jul 17 20:44:58.980583 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:58.980338    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8_odh-ai-gateway-infra(661abf74-7e3e-48d2-8088-45da1b382787)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" podUID="661abf74-7e3e-48d2-8088-45da1b382787"
Jul 17 20:44:58.981386 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:58.981360    2572 generic.go:356] "Generic (PLEG): container finished" podID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerID="e46954c8cd693d2223501cfc696db987166236fdb563d3ca85929c4d843781c0" exitCode=6
Jul 17 20:44:58.981508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:58.981439    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" event={"ID":"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5","Type":"ContainerDied","Data":"e46954c8cd693d2223501cfc696db987166236fdb563d3ca85929c4d843781c0"}
Jul 17 20:44:58.981508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:58.981484    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" event={"ID":"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5","Type":"ContainerStarted","Data":"5d57ad685ff26f40dbc261f950bce9ad24f7526a21d74bcd05bcd1d222d4734a"}
Jul 17 20:44:58.981617 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:58.981576    2572 scope.go:117] "RemoveContainer" containerID="e46954c8cd693d2223501cfc696db987166236fdb563d3ca85929c4d843781c0"
Jul 17 20:44:59.987494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:59.987464    2572 generic.go:356] "Generic (PLEG): container finished" podID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerID="8172bda82f01b7a0118a80cebf05f5291953104346ce06405ae214d5f48b82ab" exitCode=6
Jul 17 20:44:59.987938 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:59.987538    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" event={"ID":"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5","Type":"ContainerDied","Data":"8172bda82f01b7a0118a80cebf05f5291953104346ce06405ae214d5f48b82ab"}
Jul 17 20:44:59.987938 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:59.987613    2572 scope.go:117] "RemoveContainer" containerID="e46954c8cd693d2223501cfc696db987166236fdb563d3ca85929c4d843781c0"
Jul 17 20:44:59.987938 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:44:59.987842    2572 scope.go:117] "RemoveContainer" containerID="8172bda82f01b7a0118a80cebf05f5291953104346ce06405ae214d5f48b82ab"
Jul 17 20:44:59.988159 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:44:59.988028    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb_odh-ai-gateway-infra(533f7e34-c6ae-4ce4-a38c-ce681cff1ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5"
Jul 17 20:45:00.129880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.129850    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"]
Jul 17 20:45:00.134397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.134374    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6"]
Jul 17 20:45:00.134533 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.134490    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"
Jul 17 20:45:00.137552 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.137537    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6"
Jul 17 20:45:00.142703 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.142642    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6"]
Jul 17 20:45:00.144795 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.144765    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"]
Jul 17 20:45:00.147092 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.147038    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"
Jul 17 20:45:00.154493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.154473    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6"
Jul 17 20:45:00.298207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.298173    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"]
Jul 17 20:45:00.301372 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:00.301336    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ba7a16_6378_45c8_ba06_ca5faad2dfaf.slice/crio-25b1abf11df2520c0d5290064fb39c84e81a478d7fe41abe0e08f9b00a5a6b47 WatchSource:0}: Error finding container 25b1abf11df2520c0d5290064fb39c84e81a478d7fe41abe0e08f9b00a5a6b47: Status 404 returned error can't find the container with id 25b1abf11df2520c0d5290064fb39c84e81a478d7fe41abe0e08f9b00a5a6b47
Jul 17 20:45:00.324312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.324291    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6"]
Jul 17 20:45:00.326796 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:00.326772    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d6b614_00c8_46b1_baaa_0a5a159bdeed.slice/crio-ef3aefb06de86e976ec9d86d28dc2573da271e91ef5523d697b804507b5888b2 WatchSource:0}: Error finding container ef3aefb06de86e976ec9d86d28dc2573da271e91ef5523d697b804507b5888b2: Status 404 returned error can't find the container with id ef3aefb06de86e976ec9d86d28dc2573da271e91ef5523d697b804507b5888b2
Jul 17 20:45:00.994115 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.994078    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6" event={"ID":"76d6b614-00c8-46b1-baaa-0a5a159bdeed","Type":"ContainerStarted","Data":"6c5fb6eac26fe7e2b8cff9ee98abcb56d744886167618fd1275e147a954caf25"}
Jul 17 20:45:00.994591 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.994134    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6" event={"ID":"76d6b614-00c8-46b1-baaa-0a5a159bdeed","Type":"ContainerStarted","Data":"ef3aefb06de86e976ec9d86d28dc2573da271e91ef5523d697b804507b5888b2"}
Jul 17 20:45:00.995519 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.995490    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2" event={"ID":"96ba7a16-6378-45c8-ba06-ca5faad2dfaf","Type":"ContainerStarted","Data":"194456adff718cc9879c44ddbcbc42d1d1f42909512fe454b0f2a21abc169f0d"}
Jul 17 20:45:00.995629 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.995540    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2" event={"ID":"96ba7a16-6378-45c8-ba06-ca5faad2dfaf","Type":"ContainerStarted","Data":"25b1abf11df2520c0d5290064fb39c84e81a478d7fe41abe0e08f9b00a5a6b47"}
Jul 17 20:45:00.996991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:00.996971    2572 scope.go:117] "RemoveContainer" containerID="8172bda82f01b7a0118a80cebf05f5291953104346ce06405ae214d5f48b82ab"
Jul 17 20:45:00.997177 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:00.997156    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb_odh-ai-gateway-infra(533f7e34-c6ae-4ce4-a38c-ce681cff1ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5"
Jul 17 20:45:01.008880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:01.008838    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6" podStartSLOduration=1.008824915 podStartE2EDuration="1.008824915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:45:01.00724618 +0000 UTC m=+2935.701222322" watchObservedRunningTime="2026-07-17 20:45:01.008824915 +0000 UTC m=+2935.702801052"
Jul 17 20:45:01.037275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:01.037219    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2" podStartSLOduration=1.03720163 podStartE2EDuration="1.03720163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:45:01.036463409 +0000 UTC m=+2935.730439554" watchObservedRunningTime="2026-07-17 20:45:01.03720163 +0000 UTC m=+2935.731177776"
Jul 17 20:45:01.587915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:01.587834    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:45:02.003158 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.003123    2572 generic.go:356] "Generic (PLEG): container finished" podID="76d6b614-00c8-46b1-baaa-0a5a159bdeed" containerID="6c5fb6eac26fe7e2b8cff9ee98abcb56d744886167618fd1275e147a954caf25" exitCode=0
Jul 17 20:45:02.003617 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.003209    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6" event={"ID":"76d6b614-00c8-46b1-baaa-0a5a159bdeed","Type":"ContainerDied","Data":"6c5fb6eac26fe7e2b8cff9ee98abcb56d744886167618fd1275e147a954caf25"}
Jul 17 20:45:02.004740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.004717    2572 generic.go:356] "Generic (PLEG): container finished" podID="96ba7a16-6378-45c8-ba06-ca5faad2dfaf" containerID="194456adff718cc9879c44ddbcbc42d1d1f42909512fe454b0f2a21abc169f0d" exitCode=0
Jul 17 20:45:02.004844 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.004794    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2" event={"ID":"96ba7a16-6378-45c8-ba06-ca5faad2dfaf","Type":"ContainerDied","Data":"194456adff718cc9879c44ddbcbc42d1d1f42909512fe454b0f2a21abc169f0d"}
Jul 17 20:45:02.636310 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.636274    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"]
Jul 17 20:45:02.639773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.639749    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"
Jul 17 20:45:02.647326 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.647275    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"]
Jul 17 20:45:02.736384 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.736344    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931c3b3-e453-448d-905c-5879a4a08812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm\" (UID: \"5931c3b3-e453-448d-905c-5879a4a08812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"
Jul 17 20:45:02.837111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.837075    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931c3b3-e453-448d-905c-5879a4a08812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm\" (UID: \"5931c3b3-e453-448d-905c-5879a4a08812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"
Jul 17 20:45:02.837687 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.837666    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931c3b3-e453-448d-905c-5879a4a08812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm\" (UID: \"5931c3b3-e453-448d-905c-5879a4a08812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"
Jul 17 20:45:02.951774 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:02.951698    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"
Jul 17 20:45:03.098319 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:03.098292    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"]
Jul 17 20:45:03.098984 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:03.098958    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5931c3b3_e453_448d_905c_5879a4a08812.slice/crio-a41150b9c5111734717535a8ae6a4bbbb801f12dae3ee35fc9c5f82722efa9f8 WatchSource:0}: Error finding container a41150b9c5111734717535a8ae6a4bbbb801f12dae3ee35fc9c5f82722efa9f8: Status 404 returned error can't find the container with id a41150b9c5111734717535a8ae6a4bbbb801f12dae3ee35fc9c5f82722efa9f8
Jul 17 20:45:03.182020 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:03.181999    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6"
Jul 17 20:45:03.183967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:03.183951    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"
Jul 17 20:45:04.018564 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.018542    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"
Jul 17 20:45:04.018712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.018589    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2" event={"ID":"96ba7a16-6378-45c8-ba06-ca5faad2dfaf","Type":"ContainerDied","Data":"25b1abf11df2520c0d5290064fb39c84e81a478d7fe41abe0e08f9b00a5a6b47"}
Jul 17 20:45:04.018712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.018637    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25b1abf11df2520c0d5290064fb39c84e81a478d7fe41abe0e08f9b00a5a6b47"
Jul 17 20:45:04.020129 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.020105    2572 generic.go:356] "Generic (PLEG): container finished" podID="5931c3b3-e453-448d-905c-5879a4a08812" containerID="a597a56c4f81d1cf633d510fb32dfff2a792520e8b22716f1b43fa8f226c037b" exitCode=0
Jul 17 20:45:04.020262 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.020216    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm" event={"ID":"5931c3b3-e453-448d-905c-5879a4a08812","Type":"ContainerDied","Data":"a597a56c4f81d1cf633d510fb32dfff2a792520e8b22716f1b43fa8f226c037b"}
Jul 17 20:45:04.020333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.020261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm" event={"ID":"5931c3b3-e453-448d-905c-5879a4a08812","Type":"ContainerStarted","Data":"a41150b9c5111734717535a8ae6a4bbbb801f12dae3ee35fc9c5f82722efa9f8"}
Jul 17 20:45:04.021425 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.021399    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6" event={"ID":"76d6b614-00c8-46b1-baaa-0a5a159bdeed","Type":"ContainerDied","Data":"ef3aefb06de86e976ec9d86d28dc2573da271e91ef5523d697b804507b5888b2"}
Jul 17 20:45:04.021508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.021429    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3aefb06de86e976ec9d86d28dc2573da271e91ef5523d697b804507b5888b2"
Jul 17 20:45:04.021508 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:04.021405    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738685-fxbx6"
Jul 17 20:45:05.165210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:05.165186    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"
Jul 17 20:45:05.257195 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:05.257165    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931c3b3-e453-448d-905c-5879a4a08812-maas-api-service-ca\") pod \"5931c3b3-e453-448d-905c-5879a4a08812\" (UID: \"5931c3b3-e453-448d-905c-5879a4a08812\") "
Jul 17 20:45:05.257537 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:05.257515    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5931c3b3-e453-448d-905c-5879a4a08812-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5931c3b3-e453-448d-905c-5879a4a08812" (UID: "5931c3b3-e453-448d-905c-5879a4a08812"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:05.357763 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:05.357678    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931c3b3-e453-448d-905c-5879a4a08812-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:06.036493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:06.036466    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm"
Jul 17 20:45:06.036493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:06.036486    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm" event={"ID":"5931c3b3-e453-448d-905c-5879a4a08812","Type":"ContainerDied","Data":"a41150b9c5111734717535a8ae6a4bbbb801f12dae3ee35fc9c5f82722efa9f8"}
Jul 17 20:45:06.036694 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:06.036551    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41150b9c5111734717535a8ae6a4bbbb801f12dae3ee35fc9c5f82722efa9f8"
Jul 17 20:45:06.847756 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:06.847726    2572 scope.go:117] "RemoveContainer" containerID="4c7d7154929393bde20cad288e4041dbd252aac44e6a6d3a4f843b2a7baf8416"
Jul 17 20:45:08.047523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:08.047491    2572 generic.go:356] "Generic (PLEG): container finished" podID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerID="06dcb9d76a34099b998865c80d9b05923d54ace7899e970e462dbac77f267758" exitCode=6
Jul 17 20:45:08.047908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:08.047566    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" event={"ID":"69055cdc-8732-4eb9-aad0-077f61c76d0f","Type":"ContainerDied","Data":"06dcb9d76a34099b998865c80d9b05923d54ace7899e970e462dbac77f267758"}
Jul 17 20:45:08.047908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:08.047627    2572 scope.go:117] "RemoveContainer" containerID="4c7d7154929393bde20cad288e4041dbd252aac44e6a6d3a4f843b2a7baf8416"
Jul 17 20:45:08.047908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:08.047905    2572 scope.go:117] "RemoveContainer" containerID="06dcb9d76a34099b998865c80d9b05923d54ace7899e970e462dbac77f267758"
Jul 17 20:45:08.048132 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:08.048106    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf_odh-ai-gateway-infra(69055cdc-8732-4eb9-aad0-077f61c76d0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f"
Jul 17 20:45:09.072679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:09.072647    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"]
Jul 17 20:45:09.196159 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:09.196136    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"
Jul 17 20:45:09.292195 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:09.292160    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69055cdc-8732-4eb9-aad0-077f61c76d0f-maas-api-service-ca\") pod \"69055cdc-8732-4eb9-aad0-077f61c76d0f\" (UID: \"69055cdc-8732-4eb9-aad0-077f61c76d0f\") "
Jul 17 20:45:09.292518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:09.292495    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69055cdc-8732-4eb9-aad0-077f61c76d0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69055cdc-8732-4eb9-aad0-077f61c76d0f" (UID: "69055cdc-8732-4eb9-aad0-077f61c76d0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:09.393463 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:09.393378    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69055cdc-8732-4eb9-aad0-077f61c76d0f-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:10.060356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:10.060323    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf" event={"ID":"69055cdc-8732-4eb9-aad0-077f61c76d0f","Type":"ContainerDied","Data":"4bf356f2f43c24e1d61166af338c333c2bbb42009790bc2c17307207bc782c5e"}
Jul 17 20:45:10.060356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:10.060355    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"
Jul 17 20:45:10.060619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:10.060380    2572 scope.go:117] "RemoveContainer" containerID="06dcb9d76a34099b998865c80d9b05923d54ace7899e970e462dbac77f267758"
Jul 17 20:45:10.077103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:10.077074    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"]
Jul 17 20:45:10.085347 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:10.085323    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-99hjf"]
Jul 17 20:45:10.848063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:10.848019    2572 scope.go:117] "RemoveContainer" containerID="c897e68a3c65ed0c9989e860a7069f0e164ea131bbe3eab531bf83e8d0136fe9"
Jul 17 20:45:11.530863 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.530835    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"]
Jul 17 20:45:11.531195 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531183    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.531195 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531195    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531208    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d6b614-00c8-46b1-baaa-0a5a159bdeed" containerName="cleanup"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531214    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d6b614-00c8-46b1-baaa-0a5a159bdeed" containerName="cleanup"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531222    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ba7a16-6378-45c8-ba06-ca5faad2dfaf" containerName="cleanup"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531229    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ba7a16-6378-45c8-ba06-ca5faad2dfaf" containerName="cleanup"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531239    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531245    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531251    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5931c3b3-e453-448d-905c-5879a4a08812" containerName="revoke-keys"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531256    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5931c3b3-e453-448d-905c-5879a4a08812" containerName="revoke-keys"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531263    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.531265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531268    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.531540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531319    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.531540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531327    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d6b614-00c8-46b1-baaa-0a5a159bdeed" containerName="cleanup"
Jul 17 20:45:11.531540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531332    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ba7a16-6378-45c8-ba06-ca5faad2dfaf" containerName="cleanup"
Jul 17 20:45:11.531540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531339    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5931c3b3-e453-448d-905c-5879a4a08812" containerName="revoke-keys"
Jul 17 20:45:11.531540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.531345    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:11.534269 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.534249    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"
Jul 17 20:45:11.541703 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.541678    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"]
Jul 17 20:45:11.610238 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.610208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e610d96-a6d7-419d-9e1d-496b46728506-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-66cz6\" (UID: \"2e610d96-a6d7-419d-9e1d-496b46728506\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"
Jul 17 20:45:11.711478 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.711432    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e610d96-a6d7-419d-9e1d-496b46728506-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-66cz6\" (UID: \"2e610d96-a6d7-419d-9e1d-496b46728506\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"
Jul 17 20:45:11.712084 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.712036    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e610d96-a6d7-419d-9e1d-496b46728506-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-66cz6\" (UID: \"2e610d96-a6d7-419d-9e1d-496b46728506\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"
Jul 17 20:45:11.853888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.853805    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" path="/var/lib/kubelet/pods/69055cdc-8732-4eb9-aad0-077f61c76d0f/volumes"
Jul 17 20:45:11.854411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.854397    2572 scope.go:117] "RemoveContainer" containerID="8172bda82f01b7a0118a80cebf05f5291953104346ce06405ae214d5f48b82ab"
Jul 17 20:45:11.893834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:11.893805    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"
Jul 17 20:45:12.027408 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:12.027388    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"]
Jul 17 20:45:12.039604 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:12.039574    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e610d96_a6d7_419d_9e1d_496b46728506.slice/crio-6720bcc3badd0a190099dd1045e72b5bfe4f464ae7c5393e8e7443205b4cca6a WatchSource:0}: Error finding container 6720bcc3badd0a190099dd1045e72b5bfe4f464ae7c5393e8e7443205b4cca6a: Status 404 returned error can't find the container with id 6720bcc3badd0a190099dd1045e72b5bfe4f464ae7c5393e8e7443205b4cca6a
Jul 17 20:45:12.072728 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:12.072701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" event={"ID":"2e610d96-a6d7-419d-9e1d-496b46728506","Type":"ContainerStarted","Data":"6720bcc3badd0a190099dd1045e72b5bfe4f464ae7c5393e8e7443205b4cca6a"}
Jul 17 20:45:12.074735 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:12.074708    2572 generic.go:356] "Generic (PLEG): container finished" podID="661abf74-7e3e-48d2-8088-45da1b382787" containerID="6fa6c8fa78b497e748aa159beefc6e2fc2151c9323c5b5a76216650753d21817" exitCode=6
Jul 17 20:45:12.074818 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:12.074740    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" event={"ID":"661abf74-7e3e-48d2-8088-45da1b382787","Type":"ContainerDied","Data":"6fa6c8fa78b497e748aa159beefc6e2fc2151c9323c5b5a76216650753d21817"}
Jul 17 20:45:12.074818 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:12.074782    2572 scope.go:117] "RemoveContainer" containerID="c897e68a3c65ed0c9989e860a7069f0e164ea131bbe3eab531bf83e8d0136fe9"
Jul 17 20:45:12.075078 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:12.075035    2572 scope.go:117] "RemoveContainer" containerID="6fa6c8fa78b497e748aa159beefc6e2fc2151c9323c5b5a76216650753d21817"
Jul 17 20:45:12.075266 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:12.075246    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8_odh-ai-gateway-infra(661abf74-7e3e-48d2-8088-45da1b382787)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" podUID="661abf74-7e3e-48d2-8088-45da1b382787"
Jul 17 20:45:13.005725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.005692    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"]
Jul 17 20:45:13.006098 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.005913    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5" podUID="ae872c5f-099e-4665-895e-76d0fae17e42" containerName="maas-api" containerID="cri-o://04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65" gracePeriod=30
Jul 17 20:45:13.067597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.067563    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"]
Jul 17 20:45:13.070557 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.070527    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3a4159-29738685-t48g2"]
Jul 17 20:45:13.080425 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.080403    2572 generic.go:356] "Generic (PLEG): container finished" podID="2e610d96-a6d7-419d-9e1d-496b46728506" containerID="f4e384de3ce66128b3043fa52deb50757c129a71c3399e38a226d5c3cb98678f" exitCode=6
Jul 17 20:45:13.080545 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.080471    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" event={"ID":"2e610d96-a6d7-419d-9e1d-496b46728506","Type":"ContainerDied","Data":"f4e384de3ce66128b3043fa52deb50757c129a71c3399e38a226d5c3cb98678f"}
Jul 17 20:45:13.080698 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.080679    2572 scope.go:117] "RemoveContainer" containerID="f4e384de3ce66128b3043fa52deb50757c129a71c3399e38a226d5c3cb98678f"
Jul 17 20:45:13.083919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.083898    2572 generic.go:356] "Generic (PLEG): container finished" podID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerID="08ed18c62c4e517628341a285b0c617e573108600e161b794cf99df2be55a86b" exitCode=6
Jul 17 20:45:13.084010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.083972    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" event={"ID":"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5","Type":"ContainerDied","Data":"08ed18c62c4e517628341a285b0c617e573108600e161b794cf99df2be55a86b"}
Jul 17 20:45:13.084070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.084022    2572 scope.go:117] "RemoveContainer" containerID="8172bda82f01b7a0118a80cebf05f5291953104346ce06405ae214d5f48b82ab"
Jul 17 20:45:13.084297 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.084279    2572 scope.go:117] "RemoveContainer" containerID="08ed18c62c4e517628341a285b0c617e573108600e161b794cf99df2be55a86b"
Jul 17 20:45:13.084456 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:13.084438    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb_odh-ai-gateway-infra(533f7e34-c6ae-4ce4-a38c-ce681cff1ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5"
Jul 17 20:45:13.119130 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.119105    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"]
Jul 17 20:45:13.272540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.272478    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"
Jul 17 20:45:13.283415 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.283391    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:45:13.428757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.428722    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf87g\" (UniqueName: \"kubernetes.io/projected/ae872c5f-099e-4665-895e-76d0fae17e42-kube-api-access-nf87g\") pod \"ae872c5f-099e-4665-895e-76d0fae17e42\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") "
Jul 17 20:45:13.428913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.428773    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661abf74-7e3e-48d2-8088-45da1b382787-maas-api-service-ca\") pod \"661abf74-7e3e-48d2-8088-45da1b382787\" (UID: \"661abf74-7e3e-48d2-8088-45da1b382787\") "
Jul 17 20:45:13.428913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.428809    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae872c5f-099e-4665-895e-76d0fae17e42-openshift-service-ca\") pod \"ae872c5f-099e-4665-895e-76d0fae17e42\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") "
Jul 17 20:45:13.428913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.428838    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae872c5f-099e-4665-895e-76d0fae17e42-maas-api-tls\") pod \"ae872c5f-099e-4665-895e-76d0fae17e42\" (UID: \"ae872c5f-099e-4665-895e-76d0fae17e42\") "
Jul 17 20:45:13.429185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.429156    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae872c5f-099e-4665-895e-76d0fae17e42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae872c5f-099e-4665-895e-76d0fae17e42" (UID: "ae872c5f-099e-4665-895e-76d0fae17e42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:13.429272 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.429194    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661abf74-7e3e-48d2-8088-45da1b382787-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "661abf74-7e3e-48d2-8088-45da1b382787" (UID: "661abf74-7e3e-48d2-8088-45da1b382787"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:13.430883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.430853    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae872c5f-099e-4665-895e-76d0fae17e42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae872c5f-099e-4665-895e-76d0fae17e42" (UID: "ae872c5f-099e-4665-895e-76d0fae17e42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 20:45:13.430990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.430875    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae872c5f-099e-4665-895e-76d0fae17e42-kube-api-access-nf87g" (OuterVolumeSpecName: "kube-api-access-nf87g") pod "ae872c5f-099e-4665-895e-76d0fae17e42" (UID: "ae872c5f-099e-4665-895e-76d0fae17e42"). InnerVolumeSpecName "kube-api-access-nf87g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:45:13.530489 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.530464    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf87g\" (UniqueName: \"kubernetes.io/projected/ae872c5f-099e-4665-895e-76d0fae17e42-kube-api-access-nf87g\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:13.530489 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.530488    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661abf74-7e3e-48d2-8088-45da1b382787-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:13.530650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.530499    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae872c5f-099e-4665-895e-76d0fae17e42-openshift-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:13.530650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.530510    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae872c5f-099e-4665-895e-76d0fae17e42-maas-api-tls\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:13.853652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:13.853571    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ba7a16-6378-45c8-ba06-ca5faad2dfaf" path="/var/lib/kubelet/pods/96ba7a16-6378-45c8-ba06-ca5faad2dfaf/volumes"
Jul 17 20:45:14.090633 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.090593    2572 generic.go:356] "Generic (PLEG): container finished" podID="ae872c5f-099e-4665-895e-76d0fae17e42" containerID="04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65" exitCode=0
Jul 17 20:45:14.091090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.090651    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"
Jul 17 20:45:14.091090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.090674    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5" event={"ID":"ae872c5f-099e-4665-895e-76d0fae17e42","Type":"ContainerDied","Data":"04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65"}
Jul 17 20:45:14.091090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.090721    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5" event={"ID":"ae872c5f-099e-4665-895e-76d0fae17e42","Type":"ContainerDied","Data":"8b35321726bc769dcabc0886015963c48f8ecb95c5f3d68a6fe17a751b2fe3bf"}
Jul 17 20:45:14.091090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.090744    2572 scope.go:117] "RemoveContainer" containerID="04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65"
Jul 17 20:45:14.092753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.092727    2572 generic.go:356] "Generic (PLEG): container finished" podID="2e610d96-a6d7-419d-9e1d-496b46728506" containerID="ef66d852e04287b8079850c2d3cd454524a669db9e278cd66c9aad8582b6e3b9" exitCode=6
Jul 17 20:45:14.092883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.092807    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" event={"ID":"2e610d96-a6d7-419d-9e1d-496b46728506","Type":"ContainerDied","Data":"ef66d852e04287b8079850c2d3cd454524a669db9e278cd66c9aad8582b6e3b9"}
Jul 17 20:45:14.093132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.093041    2572 scope.go:117] "RemoveContainer" containerID="ef66d852e04287b8079850c2d3cd454524a669db9e278cd66c9aad8582b6e3b9"
Jul 17 20:45:14.093309 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:14.093287    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-66cz6_odh-ai-gateway-infra(2e610d96-a6d7-419d-9e1d-496b46728506)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" podUID="2e610d96-a6d7-419d-9e1d-496b46728506"
Jul 17 20:45:14.094306 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.094290    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"
Jul 17 20:45:14.094387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.094319    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8" event={"ID":"661abf74-7e3e-48d2-8088-45da1b382787","Type":"ContainerDied","Data":"ff6f98c382d7a3164e81ef0d2bdf5ff5050c37fcfb28e5b6ec1a76272072dca0"}
Jul 17 20:45:14.100100 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.100082    2572 scope.go:117] "RemoveContainer" containerID="04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65"
Jul 17 20:45:14.100346 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:14.100321    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65\": container with ID starting with 04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65 not found: ID does not exist" containerID="04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65"
Jul 17 20:45:14.100418 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.100358    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65"} err="failed to get container status \"04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65\": rpc error: code = NotFound desc = could not find container \"04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65\": container with ID starting with 04505c92df7958e23d3edb344511be14c6a7d9b18619a996afd63da48dfcaf65 not found: ID does not exist"
Jul 17 20:45:14.100418 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.100382    2572 scope.go:117] "RemoveContainer" containerID="f4e384de3ce66128b3043fa52deb50757c129a71c3399e38a226d5c3cb98678f"
Jul 17 20:45:14.113161 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.113139    2572 scope.go:117] "RemoveContainer" containerID="6fa6c8fa78b497e748aa159beefc6e2fc2151c9323c5b5a76216650753d21817"
Jul 17 20:45:14.119936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.119910    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"]
Jul 17 20:45:14.125438 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.125416    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-tm4l8"]
Jul 17 20:45:14.156330 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.156305    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"]
Jul 17 20:45:14.159804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.159785    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"]
Jul 17 20:45:14.174109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.174081    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a4159-5cc48fdc4-2mtl5"]
Jul 17 20:45:14.296796 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.296773    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"
Jul 17 20:45:14.438528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.438438    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f7e34-c6ae-4ce4-a38c-ce681cff1ef5-maas-api-service-ca\") pod \"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5\" (UID: \"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5\") "
Jul 17 20:45:14.438859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.438828    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533f7e34-c6ae-4ce4-a38c-ce681cff1ef5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" (UID: "533f7e34-c6ae-4ce4-a38c-ce681cff1ef5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:14.539540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:14.539511    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f7e34-c6ae-4ce4-a38c-ce681cff1ef5-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:15.103067 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.103026    2572 scope.go:117] "RemoveContainer" containerID="ef66d852e04287b8079850c2d3cd454524a669db9e278cd66c9aad8582b6e3b9"
Jul 17 20:45:15.103488 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:15.103261    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-66cz6_odh-ai-gateway-infra(2e610d96-a6d7-419d-9e1d-496b46728506)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" podUID="2e610d96-a6d7-419d-9e1d-496b46728506"
Jul 17 20:45:15.104816 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.104787    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb" event={"ID":"533f7e34-c6ae-4ce4-a38c-ce681cff1ef5","Type":"ContainerDied","Data":"5d57ad685ff26f40dbc261f950bce9ad24f7526a21d74bcd05bcd1d222d4734a"}
Jul 17 20:45:15.104816 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.104816    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"
Jul 17 20:45:15.104996 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.104850    2572 scope.go:117] "RemoveContainer" containerID="08ed18c62c4e517628341a285b0c617e573108600e161b794cf99df2be55a86b"
Jul 17 20:45:15.149314 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.149276    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"]
Jul 17 20:45:15.153069 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.153021    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-zvqdb"]
Jul 17 20:45:15.853183 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.853153    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" path="/var/lib/kubelet/pods/533f7e34-c6ae-4ce4-a38c-ce681cff1ef5/volumes"
Jul 17 20:45:15.853472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.853458    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661abf74-7e3e-48d2-8088-45da1b382787" path="/var/lib/kubelet/pods/661abf74-7e3e-48d2-8088-45da1b382787/volumes"
Jul 17 20:45:15.853809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:15.853797    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae872c5f-099e-4665-895e-76d0fae17e42" path="/var/lib/kubelet/pods/ae872c5f-099e-4665-895e-76d0fae17e42/volumes"
Jul 17 20:45:17.135059 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:17.135023    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:45:29.044837 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.044745    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"]
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045193    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045212    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045223    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045232    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045248    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045256    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045270    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae872c5f-099e-4665-895e-76d0fae17e42" containerName="maas-api"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045278    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae872c5f-099e-4665-895e-76d0fae17e42" containerName="maas-api"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045298    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:29.045335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045305    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045378    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae872c5f-099e-4665-895e-76d0fae17e42" containerName="maas-api"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045392    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045400    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="69055cdc-8732-4eb9-aad0-077f61c76d0f" containerName="revoke-keys"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045411    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045419    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045428    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045437    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:29.045645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.045443    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:29.049951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.049928    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"
Jul 17 20:45:29.077867 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.077829    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"]
Jul 17 20:45:29.165248 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.165217    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b56ed59-c174-47e1-80ae-626557b7e990-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk\" (UID: \"3b56ed59-c174-47e1-80ae-626557b7e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"
Jul 17 20:45:29.266577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.266527    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b56ed59-c174-47e1-80ae-626557b7e990-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk\" (UID: \"3b56ed59-c174-47e1-80ae-626557b7e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"
Jul 17 20:45:29.267202 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.267174    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b56ed59-c174-47e1-80ae-626557b7e990-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk\" (UID: \"3b56ed59-c174-47e1-80ae-626557b7e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"
Jul 17 20:45:29.359812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.359734    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"
Jul 17 20:45:29.509151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.509129    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"]
Jul 17 20:45:29.511620 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:29.511590    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b56ed59_c174_47e1_80ae_626557b7e990.slice/crio-5a8091e30d02a4021d29e7ac3c029f53da5aff2078d440d689f215376bc0e9d7 WatchSource:0}: Error finding container 5a8091e30d02a4021d29e7ac3c029f53da5aff2078d440d689f215376bc0e9d7: Status 404 returned error can't find the container with id 5a8091e30d02a4021d29e7ac3c029f53da5aff2078d440d689f215376bc0e9d7
Jul 17 20:45:29.848611 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:29.848579    2572 scope.go:117] "RemoveContainer" containerID="ef66d852e04287b8079850c2d3cd454524a669db9e278cd66c9aad8582b6e3b9"
Jul 17 20:45:30.187860 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:30.187824    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" event={"ID":"2e610d96-a6d7-419d-9e1d-496b46728506","Type":"ContainerStarted","Data":"73fecdc668c352c4fc0476638978eb8517364940cc311e482755f1336725fa58"}
Jul 17 20:45:30.189314 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:30.189286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk" event={"ID":"3b56ed59-c174-47e1-80ae-626557b7e990","Type":"ContainerStarted","Data":"c09a27155c8c4b662389205adfaf2696e2d12fa4907647cc76683d0438a2db42"}
Jul 17 20:45:30.189458 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:30.189326    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk" event={"ID":"3b56ed59-c174-47e1-80ae-626557b7e990","Type":"ContainerStarted","Data":"5a8091e30d02a4021d29e7ac3c029f53da5aff2078d440d689f215376bc0e9d7"}
Jul 17 20:45:30.271881 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:30.271828    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk" podStartSLOduration=1.271810334 podStartE2EDuration="1.271810334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:45:30.260986658 +0000 UTC m=+2964.954962805" watchObservedRunningTime="2026-07-17 20:45:30.271810334 +0000 UTC m=+2964.965786545"
Jul 17 20:45:30.272088 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:30.271923    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" podStartSLOduration=19.271917379 podStartE2EDuration="19.271917379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:45:30.218302045 +0000 UTC m=+2964.912278191" watchObservedRunningTime="2026-07-17 20:45:30.271917379 +0000 UTC m=+2964.965893525"
Jul 17 20:45:31.196014 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:31.195979    2572 generic.go:356] "Generic (PLEG): container finished" podID="2e610d96-a6d7-419d-9e1d-496b46728506" containerID="73fecdc668c352c4fc0476638978eb8517364940cc311e482755f1336725fa58" exitCode=6
Jul 17 20:45:31.196484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:31.196066    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" event={"ID":"2e610d96-a6d7-419d-9e1d-496b46728506","Type":"ContainerDied","Data":"73fecdc668c352c4fc0476638978eb8517364940cc311e482755f1336725fa58"}
Jul 17 20:45:31.196484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:31.196127    2572 scope.go:117] "RemoveContainer" containerID="ef66d852e04287b8079850c2d3cd454524a669db9e278cd66c9aad8582b6e3b9"
Jul 17 20:45:31.196484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:31.196395    2572 scope.go:117] "RemoveContainer" containerID="73fecdc668c352c4fc0476638978eb8517364940cc311e482755f1336725fa58"
Jul 17 20:45:31.196653 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:31.196628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-66cz6_odh-ai-gateway-infra(2e610d96-a6d7-419d-9e1d-496b46728506)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" podUID="2e610d96-a6d7-419d-9e1d-496b46728506"
Jul 17 20:45:31.197649 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:31.197625    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b56ed59-c174-47e1-80ae-626557b7e990" containerID="c09a27155c8c4b662389205adfaf2696e2d12fa4907647cc76683d0438a2db42" exitCode=0
Jul 17 20:45:31.197743 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:31.197701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk" event={"ID":"3b56ed59-c174-47e1-80ae-626557b7e990","Type":"ContainerDied","Data":"c09a27155c8c4b662389205adfaf2696e2d12fa4907647cc76683d0438a2db42"}
Jul 17 20:45:31.231151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:31.231111    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"]
Jul 17 20:45:32.352235 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.352194    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"
Jul 17 20:45:32.354169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.354152    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"
Jul 17 20:45:32.390862 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.390829    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b56ed59-c174-47e1-80ae-626557b7e990-maas-api-service-ca\") pod \"3b56ed59-c174-47e1-80ae-626557b7e990\" (UID: \"3b56ed59-c174-47e1-80ae-626557b7e990\") "
Jul 17 20:45:32.391010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.390891    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e610d96-a6d7-419d-9e1d-496b46728506-maas-api-service-ca\") pod \"2e610d96-a6d7-419d-9e1d-496b46728506\" (UID: \"2e610d96-a6d7-419d-9e1d-496b46728506\") "
Jul 17 20:45:32.391261 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.391237    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b56ed59-c174-47e1-80ae-626557b7e990-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b56ed59-c174-47e1-80ae-626557b7e990" (UID: "3b56ed59-c174-47e1-80ae-626557b7e990"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:32.391309 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.391237    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e610d96-a6d7-419d-9e1d-496b46728506-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e610d96-a6d7-419d-9e1d-496b46728506" (UID: "2e610d96-a6d7-419d-9e1d-496b46728506"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:32.491939 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.491904    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b56ed59-c174-47e1-80ae-626557b7e990-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:32.491939 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:32.491933    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e610d96-a6d7-419d-9e1d-496b46728506-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:33.209673 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.209647    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"
Jul 17 20:45:33.209880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.209676    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6" event={"ID":"2e610d96-a6d7-419d-9e1d-496b46728506","Type":"ContainerDied","Data":"6720bcc3badd0a190099dd1045e72b5bfe4f464ae7c5393e8e7443205b4cca6a"}
Jul 17 20:45:33.209880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.209741    2572 scope.go:117] "RemoveContainer" containerID="73fecdc668c352c4fc0476638978eb8517364940cc311e482755f1336725fa58"
Jul 17 20:45:33.211285 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.211261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk" event={"ID":"3b56ed59-c174-47e1-80ae-626557b7e990","Type":"ContainerDied","Data":"5a8091e30d02a4021d29e7ac3c029f53da5aff2078d440d689f215376bc0e9d7"}
Jul 17 20:45:33.211376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.211304    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a8091e30d02a4021d29e7ac3c029f53da5aff2078d440d689f215376bc0e9d7"
Jul 17 20:45:33.211376 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.211273    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk"
Jul 17 20:45:33.241281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.241256    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"]
Jul 17 20:45:33.249909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.249888    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-66cz6"]
Jul 17 20:45:33.852873 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:33.852844    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" path="/var/lib/kubelet/pods/2e610d96-a6d7-419d-9e1d-496b46728506/volumes"
Jul 17 20:45:39.690194 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:39.690161    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"]
Jul 17 20:45:39.690563 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:39.690364    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw" podUID="f573bda4-3bda-405c-8c59-c6923f064c99" containerName="maas-api" containerID="cri-o://6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a" gracePeriod=30
Jul 17 20:45:39.942387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:39.942328    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:45:40.042301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.042271    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls\") pod \"f573bda4-3bda-405c-8c59-c6923f064c99\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") "
Jul 17 20:45:40.042488 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.042309    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zsrr\" (UniqueName: \"kubernetes.io/projected/f573bda4-3bda-405c-8c59-c6923f064c99-kube-api-access-7zsrr\") pod \"f573bda4-3bda-405c-8c59-c6923f064c99\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") "
Jul 17 20:45:40.042488 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.042350    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f573bda4-3bda-405c-8c59-c6923f064c99-openshift-service-ca\") pod \"f573bda4-3bda-405c-8c59-c6923f064c99\" (UID: \"f573bda4-3bda-405c-8c59-c6923f064c99\") "
Jul 17 20:45:40.042786 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.042752    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f573bda4-3bda-405c-8c59-c6923f064c99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f573bda4-3bda-405c-8c59-c6923f064c99" (UID: "f573bda4-3bda-405c-8c59-c6923f064c99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:45:40.044442 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.044405    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f573bda4-3bda-405c-8c59-c6923f064c99-kube-api-access-7zsrr" (OuterVolumeSpecName: "kube-api-access-7zsrr") pod "f573bda4-3bda-405c-8c59-c6923f064c99" (UID: "f573bda4-3bda-405c-8c59-c6923f064c99"). InnerVolumeSpecName "kube-api-access-7zsrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:45:40.044550 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.044478    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f573bda4-3bda-405c-8c59-c6923f064c99" (UID: "f573bda4-3bda-405c-8c59-c6923f064c99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 20:45:40.142917 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.142880    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zsrr\" (UniqueName: \"kubernetes.io/projected/f573bda4-3bda-405c-8c59-c6923f064c99-kube-api-access-7zsrr\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:40.142917 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.142910    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f573bda4-3bda-405c-8c59-c6923f064c99-openshift-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:40.142917 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.142923    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f573bda4-3bda-405c-8c59-c6923f064c99-maas-api-tls\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:45:40.244142 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.244103    2572 generic.go:356] "Generic (PLEG): container finished" podID="f573bda4-3bda-405c-8c59-c6923f064c99" containerID="6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a" exitCode=0
Jul 17 20:45:40.244142 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.244148    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"
Jul 17 20:45:40.244343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.244185    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw" event={"ID":"f573bda4-3bda-405c-8c59-c6923f064c99","Type":"ContainerDied","Data":"6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a"}
Jul 17 20:45:40.244343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.244240    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw" event={"ID":"f573bda4-3bda-405c-8c59-c6923f064c99","Type":"ContainerDied","Data":"042c1d82c1844d09cbec9aefa207982213bb5ccd65b11a15eeaad29f0e3bf237"}
Jul 17 20:45:40.244343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.244261    2572 scope.go:117] "RemoveContainer" containerID="6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a"
Jul 17 20:45:40.252791 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.252773    2572 scope.go:117] "RemoveContainer" containerID="6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a"
Jul 17 20:45:40.253028 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:40.253011    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a\": container with ID starting with 6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a not found: ID does not exist" containerID="6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a"
Jul 17 20:45:40.253110 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.253036    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a"} err="failed to get container status \"6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a\": rpc error: code = NotFound desc = could not find container \"6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a\": container with ID starting with 6f5c986093ffb4deb4fddae301ee940130522d973996e9c3b9bd046756da6f4a not found: ID does not exist"
Jul 17 20:45:40.272654 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.272628    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"]
Jul 17 20:45:40.291924 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:40.291893    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad0e32-5f647d5f7f-99bfw"]
Jul 17 20:45:41.853610 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:41.853574    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f573bda4-3bda-405c-8c59-c6923f064c99" path="/var/lib/kubelet/pods/f573bda4-3bda-405c-8c59-c6923f064c99/volumes"
Jul 17 20:45:42.611590 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:42.611554    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:45:45.644267 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644233    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"]
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644531    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644542    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="661abf74-7e3e-48d2-8088-45da1b382787" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644552    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f573bda4-3bda-405c-8c59-c6923f064c99" containerName="maas-api"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644558    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f573bda4-3bda-405c-8c59-c6923f064c99" containerName="maas-api"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644565    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644570    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644577    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644583    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644589    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644595    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="533f7e34-c6ae-4ce4-a38c-ce681cff1ef5" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644607    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b56ed59-c174-47e1-80ae-626557b7e990" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644613    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b56ed59-c174-47e1-80ae-626557b7e990" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644620    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644626    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.644669 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644674    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f573bda4-3bda-405c-8c59-c6923f064c99" containerName="maas-api"
Jul 17 20:45:45.645198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644681    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.645198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644687    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.645198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644693    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b56ed59-c174-47e1-80ae-626557b7e990" containerName="revoke-keys"
Jul 17 20:45:45.645198 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.644699    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e610d96-a6d7-419d-9e1d-496b46728506" containerName="revoke-keys"
Jul 17 20:45:45.649128 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.649109    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"
Jul 17 20:45:45.652773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.652750    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:45:45.652887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.652841    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:45:45.665476 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.665448    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"]
Jul 17 20:45:45.688888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.688859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a4d85-6b69-4747-9616-7414d9990d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-snv9n\" (UID: \"d66a4d85-6b69-4747-9616-7414d9990d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"
Jul 17 20:45:45.789476 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.789443    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a4d85-6b69-4747-9616-7414d9990d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-snv9n\" (UID: \"d66a4d85-6b69-4747-9616-7414d9990d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"
Jul 17 20:45:45.790034 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.790014    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a4d85-6b69-4747-9616-7414d9990d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-snv9n\" (UID: \"d66a4d85-6b69-4747-9616-7414d9990d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"
Jul 17 20:45:45.958648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:45.958561    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"
Jul 17 20:45:46.094647 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:46.094624    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"]
Jul 17 20:45:46.096775 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:46.096743    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66a4d85_6b69_4747_9616_7414d9990d82.slice/crio-adc0c71ed607159912f7aaa9c7a8e3073fce6e5d66750fc2334248e7772c8fac WatchSource:0}: Error finding container adc0c71ed607159912f7aaa9c7a8e3073fce6e5d66750fc2334248e7772c8fac: Status 404 returned error can't find the container with id adc0c71ed607159912f7aaa9c7a8e3073fce6e5d66750fc2334248e7772c8fac
Jul 17 20:45:46.274623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:46.274590    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" event={"ID":"d66a4d85-6b69-4747-9616-7414d9990d82","Type":"ContainerStarted","Data":"adc0c71ed607159912f7aaa9c7a8e3073fce6e5d66750fc2334248e7772c8fac"}
Jul 17 20:45:47.280829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:47.280794    2572 generic.go:356] "Generic (PLEG): container finished" podID="d66a4d85-6b69-4747-9616-7414d9990d82" containerID="97ad0dd3d2c9284096b98facf9e55c586a4462db0d04f09cad7ec5448868a535" exitCode=6
Jul 17 20:45:47.281216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:47.280869    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" event={"ID":"d66a4d85-6b69-4747-9616-7414d9990d82","Type":"ContainerDied","Data":"97ad0dd3d2c9284096b98facf9e55c586a4462db0d04f09cad7ec5448868a535"}
Jul 17 20:45:47.281216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:47.281121    2572 scope.go:117] "RemoveContainer" containerID="97ad0dd3d2c9284096b98facf9e55c586a4462db0d04f09cad7ec5448868a535"
Jul 17 20:45:47.855634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:47.855609    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"]
Jul 17 20:45:47.858647 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:47.858632    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"
Jul 17 20:45:47.867522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:47.867496    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"]
Jul 17 20:45:47.905280 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:47.905245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd4c67f-bfc2-4f7e-a980-ea2d625035e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-ds96l\" (UID: \"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"
Jul 17 20:45:48.005690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.005658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd4c67f-bfc2-4f7e-a980-ea2d625035e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-ds96l\" (UID: \"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"
Jul 17 20:45:48.006270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.006250    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd4c67f-bfc2-4f7e-a980-ea2d625035e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-ds96l\" (UID: \"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"
Jul 17 20:45:48.196955 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.196873    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"
Jul 17 20:45:48.290494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.290402    2572 generic.go:356] "Generic (PLEG): container finished" podID="d66a4d85-6b69-4747-9616-7414d9990d82" containerID="446fc03590ebc23f0b37c1dd057280bde8ec6277ea17f160924184d2bd61fc95" exitCode=6
Jul 17 20:45:48.290494 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.290457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" event={"ID":"d66a4d85-6b69-4747-9616-7414d9990d82","Type":"ContainerDied","Data":"446fc03590ebc23f0b37c1dd057280bde8ec6277ea17f160924184d2bd61fc95"}
Jul 17 20:45:48.294481 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.294449    2572 scope.go:117] "RemoveContainer" containerID="97ad0dd3d2c9284096b98facf9e55c586a4462db0d04f09cad7ec5448868a535"
Jul 17 20:45:48.294901 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.294882    2572 scope.go:117] "RemoveContainer" containerID="446fc03590ebc23f0b37c1dd057280bde8ec6277ea17f160924184d2bd61fc95"
Jul 17 20:45:48.295219 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:48.295195    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-snv9n_odh-ai-gateway-infra(d66a4d85-6b69-4747-9616-7414d9990d82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" podUID="d66a4d85-6b69-4747-9616-7414d9990d82"
Jul 17 20:45:48.557788 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:48.557641    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"]
Jul 17 20:45:48.560448 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:48.560409    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd4c67f_bfc2_4f7e_a980_ea2d625035e4.slice/crio-858bcd60d894ede23aaa4de30a5179b056cbdf2240feac15a02a0946fe2e7f23 WatchSource:0}: Error finding container 858bcd60d894ede23aaa4de30a5179b056cbdf2240feac15a02a0946fe2e7f23: Status 404 returned error can't find the container with id 858bcd60d894ede23aaa4de30a5179b056cbdf2240feac15a02a0946fe2e7f23
Jul 17 20:45:49.303538 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.303509    2572 scope.go:117] "RemoveContainer" containerID="446fc03590ebc23f0b37c1dd057280bde8ec6277ea17f160924184d2bd61fc95"
Jul 17 20:45:49.303963 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:49.303730    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-snv9n_odh-ai-gateway-infra(d66a4d85-6b69-4747-9616-7414d9990d82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" podUID="d66a4d85-6b69-4747-9616-7414d9990d82"
Jul 17 20:45:49.304681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.304657    2572 generic.go:356] "Generic (PLEG): container finished" podID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerID="630e656f21f23e0d72ab9b4ea3964b8eed20d56dc99b340dff55b2146dd2b7e1" exitCode=6
Jul 17 20:45:49.304794 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.304738    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" event={"ID":"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4","Type":"ContainerDied","Data":"630e656f21f23e0d72ab9b4ea3964b8eed20d56dc99b340dff55b2146dd2b7e1"}
Jul 17 20:45:49.304794 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.304780    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" event={"ID":"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4","Type":"ContainerStarted","Data":"858bcd60d894ede23aaa4de30a5179b056cbdf2240feac15a02a0946fe2e7f23"}
Jul 17 20:45:49.304940 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.304929    2572 scope.go:117] "RemoveContainer" containerID="630e656f21f23e0d72ab9b4ea3964b8eed20d56dc99b340dff55b2146dd2b7e1"
Jul 17 20:45:49.645716 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.645693    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"]
Jul 17 20:45:49.648903 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.648888    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"
Jul 17 20:45:49.657834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.657806    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"]
Jul 17 20:45:49.719802 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.719768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1301b6d6-5c62-4e93-ab91-386ba2e7378f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9\" (UID: \"1301b6d6-5c62-4e93-ab91-386ba2e7378f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"
Jul 17 20:45:49.820725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.820630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1301b6d6-5c62-4e93-ab91-386ba2e7378f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9\" (UID: \"1301b6d6-5c62-4e93-ab91-386ba2e7378f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"
Jul 17 20:45:49.821301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.821280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1301b6d6-5c62-4e93-ab91-386ba2e7378f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9\" (UID: \"1301b6d6-5c62-4e93-ab91-386ba2e7378f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"
Jul 17 20:45:49.959305 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:49.959269    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"
Jul 17 20:45:50.110496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:50.110473    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"]
Jul 17 20:45:50.112253 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:45:50.112221    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1301b6d6_5c62_4e93_ab91_386ba2e7378f.slice/crio-82b37da6e358355c2a06e628497071cd5ec2fa35caa9cda8fd46a712f830aef0 WatchSource:0}: Error finding container 82b37da6e358355c2a06e628497071cd5ec2fa35caa9cda8fd46a712f830aef0: Status 404 returned error can't find the container with id 82b37da6e358355c2a06e628497071cd5ec2fa35caa9cda8fd46a712f830aef0
Jul 17 20:45:50.310858 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:50.310827    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" event={"ID":"1301b6d6-5c62-4e93-ab91-386ba2e7378f","Type":"ContainerStarted","Data":"82b37da6e358355c2a06e628497071cd5ec2fa35caa9cda8fd46a712f830aef0"}
Jul 17 20:45:50.312670 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:50.312647    2572 generic.go:356] "Generic (PLEG): container finished" podID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerID="06c9ae4671f80efcc110717e485e0e2d5c20b50f0d3b04686300f68fed6f76a0" exitCode=6
Jul 17 20:45:50.312799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:50.312693    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" event={"ID":"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4","Type":"ContainerDied","Data":"06c9ae4671f80efcc110717e485e0e2d5c20b50f0d3b04686300f68fed6f76a0"}
Jul 17 20:45:50.312799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:50.312734    2572 scope.go:117] "RemoveContainer" containerID="630e656f21f23e0d72ab9b4ea3964b8eed20d56dc99b340dff55b2146dd2b7e1"
Jul 17 20:45:50.312951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:50.312934    2572 scope.go:117] "RemoveContainer" containerID="06c9ae4671f80efcc110717e485e0e2d5c20b50f0d3b04686300f68fed6f76a0"
Jul 17 20:45:50.313167 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:50.313149    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-ds96l_odh-ai-gateway-infra(5dd4c67f-bfc2-4f7e-a980-ea2d625035e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4"
Jul 17 20:45:51.319569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:51.319535    2572 scope.go:117] "RemoveContainer" containerID="06c9ae4671f80efcc110717e485e0e2d5c20b50f0d3b04686300f68fed6f76a0"
Jul 17 20:45:51.320067 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:51.319760    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-ds96l_odh-ai-gateway-infra(5dd4c67f-bfc2-4f7e-a980-ea2d625035e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4"
Jul 17 20:45:51.320812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:51.320789    2572 generic.go:356] "Generic (PLEG): container finished" podID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerID="74f38f729568994fe734cf729665a837c1fcd2add31691b2fd504c248555e672" exitCode=6
Jul 17 20:45:51.320907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:51.320838    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" event={"ID":"1301b6d6-5c62-4e93-ab91-386ba2e7378f","Type":"ContainerDied","Data":"74f38f729568994fe734cf729665a837c1fcd2add31691b2fd504c248555e672"}
Jul 17 20:45:51.320985 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:51.320972    2572 scope.go:117] "RemoveContainer" containerID="74f38f729568994fe734cf729665a837c1fcd2add31691b2fd504c248555e672"
Jul 17 20:45:52.328101 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:52.327992    2572 generic.go:356] "Generic (PLEG): container finished" podID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerID="542b663a9ec56da3ea5d0be5513adb6876af68e35fb945ac2cc5464e660eb8b4" exitCode=6
Jul 17 20:45:52.328522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:52.328091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" event={"ID":"1301b6d6-5c62-4e93-ab91-386ba2e7378f","Type":"ContainerDied","Data":"542b663a9ec56da3ea5d0be5513adb6876af68e35fb945ac2cc5464e660eb8b4"}
Jul 17 20:45:52.328522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:52.328153    2572 scope.go:117] "RemoveContainer" containerID="74f38f729568994fe734cf729665a837c1fcd2add31691b2fd504c248555e672"
Jul 17 20:45:52.328522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:52.328329    2572 scope.go:117] "RemoveContainer" containerID="542b663a9ec56da3ea5d0be5513adb6876af68e35fb945ac2cc5464e660eb8b4"
Jul 17 20:45:52.328647 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:52.328559    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9_odh-ai-gateway-infra(1301b6d6-5c62-4e93-ab91-386ba2e7378f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f"
Jul 17 20:45:53.334437 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:53.334412    2572 scope.go:117] "RemoveContainer" containerID="542b663a9ec56da3ea5d0be5513adb6876af68e35fb945ac2cc5464e660eb8b4"
Jul 17 20:45:53.334840 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:53.334608    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9_odh-ai-gateway-infra(1301b6d6-5c62-4e93-ab91-386ba2e7378f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f"
Jul 17 20:45:59.111900 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.111867    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"]
Jul 17 20:45:59.116218 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.116202    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.119956 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.119923    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-72eefe-serving-cert\""
Jul 17 20:45:59.120084 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.119981    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 20:45:59.120335 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.120310    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mxjrb\""
Jul 17 20:45:59.135169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.135145    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"]
Jul 17 20:45:59.198944 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.198913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f67f\" (UniqueName: \"kubernetes.io/projected/ea81aa58-21b7-4b5a-b309-d31497116f65-kube-api-access-6f67f\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.199111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.198976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea81aa58-21b7-4b5a-b309-d31497116f65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.199111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.199007    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.300234 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.300195    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f67f\" (UniqueName: \"kubernetes.io/projected/ea81aa58-21b7-4b5a-b309-d31497116f65-kube-api-access-6f67f\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.300391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.300256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea81aa58-21b7-4b5a-b309-d31497116f65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.300391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.300283    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.300391 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:59.300382    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-serving-cert: secret "maas-api-e2e-shared-a-72eefe-serving-cert" not found
Jul 17 20:45:59.300540 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:45:59.300436    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls podName:ea81aa58-21b7-4b5a-b309-d31497116f65 nodeName:}" failed. No retries permitted until 2026-07-17 20:45:59.800419759 +0000 UTC m=+2994.494395883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls") pod "maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf" (UID: "ea81aa58-21b7-4b5a-b309-d31497116f65") : secret "maas-api-e2e-shared-a-72eefe-serving-cert" not found
Jul 17 20:45:59.300967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.300948    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea81aa58-21b7-4b5a-b309-d31497116f65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.316571 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.316541    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f67f\" (UniqueName: \"kubernetes.io/projected/ea81aa58-21b7-4b5a-b309-d31497116f65-kube-api-access-6f67f\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.804739 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.804700    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.807133 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.807105    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls\") pod \"maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:45:59.848195 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:45:59.848162    2572 scope.go:117] "RemoveContainer" containerID="446fc03590ebc23f0b37c1dd057280bde8ec6277ea17f160924184d2bd61fc95"
Jul 17 20:46:00.026448 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:00.026408    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:46:00.164706 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:00.164658    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"]
Jul 17 20:46:00.167423 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:46:00.167393    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea81aa58_21b7_4b5a_b309_d31497116f65.slice/crio-c3478620c6f3f8e54a8c025c74b4a10cd6fca34754133d3aaded44a866d69892 WatchSource:0}: Error finding container c3478620c6f3f8e54a8c025c74b4a10cd6fca34754133d3aaded44a866d69892: Status 404 returned error can't find the container with id c3478620c6f3f8e54a8c025c74b4a10cd6fca34754133d3aaded44a866d69892
Jul 17 20:46:00.371280 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:00.371202    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf" event={"ID":"ea81aa58-21b7-4b5a-b309-d31497116f65","Type":"ContainerStarted","Data":"c3478620c6f3f8e54a8c025c74b4a10cd6fca34754133d3aaded44a866d69892"}
Jul 17 20:46:00.372758 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:00.372737    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" event={"ID":"d66a4d85-6b69-4747-9616-7414d9990d82","Type":"ContainerStarted","Data":"84e839e948c3ce9c11ce8fe52c62c8e55041009b760e9802df136d9e6d7f1e78"}
Jul 17 20:46:01.378632 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.378597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf" event={"ID":"ea81aa58-21b7-4b5a-b309-d31497116f65","Type":"ContainerStarted","Data":"0ed2ec00b5f02ed978b826685e8bda83aa45f68b210ef3dfdbfaa4cdf217cab4"}
Jul 17 20:46:01.379129 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.378755    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:46:01.380261 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.380239    2572 generic.go:356] "Generic (PLEG): container finished" podID="d66a4d85-6b69-4747-9616-7414d9990d82" containerID="84e839e948c3ce9c11ce8fe52c62c8e55041009b760e9802df136d9e6d7f1e78" exitCode=6
Jul 17 20:46:01.380369 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.380286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" event={"ID":"d66a4d85-6b69-4747-9616-7414d9990d82","Type":"ContainerDied","Data":"84e839e948c3ce9c11ce8fe52c62c8e55041009b760e9802df136d9e6d7f1e78"}
Jul 17 20:46:01.380369 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.380321    2572 scope.go:117] "RemoveContainer" containerID="446fc03590ebc23f0b37c1dd057280bde8ec6277ea17f160924184d2bd61fc95"
Jul 17 20:46:01.380561 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.380547    2572 scope.go:117] "RemoveContainer" containerID="84e839e948c3ce9c11ce8fe52c62c8e55041009b760e9802df136d9e6d7f1e78"
Jul 17 20:46:01.380727 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:01.380709    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-snv9n_odh-ai-gateway-infra(d66a4d85-6b69-4747-9616-7414d9990d82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" podUID="d66a4d85-6b69-4747-9616-7414d9990d82"
Jul 17 20:46:01.402154 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.402112    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" podStartSLOduration=16.402097093 podStartE2EDuration="16.402097093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:00.398825757 +0000 UTC m=+2995.092801915" watchObservedRunningTime="2026-07-17 20:46:01.402097093 +0000 UTC m=+2996.096073238"
Jul 17 20:46:01.415750 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.415709    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf" podStartSLOduration=2.024392869 podStartE2EDuration="2.41569385s" totalImagesPullingTime="391.300981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:01.405390069 +0000 UTC m=+2996.099366226" watchObservedRunningTime="2026-07-17 20:46:01.41569385 +0000 UTC m=+2996.109670037"
Jul 17 20:46:01.416264 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:01.416250    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"]
Jul 17 20:46:02.517505 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:02.517484    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"
Jul 17 20:46:02.627659 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:02.627621    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a4d85-6b69-4747-9616-7414d9990d82-maas-api-service-ca\") pod \"d66a4d85-6b69-4747-9616-7414d9990d82\" (UID: \"d66a4d85-6b69-4747-9616-7414d9990d82\") "
Jul 17 20:46:02.628000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:02.627978    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66a4d85-6b69-4747-9616-7414d9990d82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d66a4d85-6b69-4747-9616-7414d9990d82" (UID: "d66a4d85-6b69-4747-9616-7414d9990d82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:46:02.729152 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:02.729110    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a4d85-6b69-4747-9616-7414d9990d82-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:46:02.848714 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:02.848683    2572 scope.go:117] "RemoveContainer" containerID="06c9ae4671f80efcc110717e485e0e2d5c20b50f0d3b04686300f68fed6f76a0"
Jul 17 20:46:03.344621 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.344530    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:46:03.393250 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.393220    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" event={"ID":"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4","Type":"ContainerStarted","Data":"09b0ae1fb6ad9e13bfbc5531e1d6c58869ed5ebcf1529ab6f3ff754ba94ebcf0"}
Jul 17 20:46:03.394312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.394291    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n" event={"ID":"d66a4d85-6b69-4747-9616-7414d9990d82","Type":"ContainerDied","Data":"adc0c71ed607159912f7aaa9c7a8e3073fce6e5d66750fc2334248e7772c8fac"}
Jul 17 20:46:03.394440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.394311    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"
Jul 17 20:46:03.394440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.394331    2572 scope.go:117] "RemoveContainer" containerID="84e839e948c3ce9c11ce8fe52c62c8e55041009b760e9802df136d9e6d7f1e78"
Jul 17 20:46:03.411369 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.411319    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" podStartSLOduration=16.411300909 podStartE2EDuration="16.411300909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:03.409319877 +0000 UTC m=+2998.103296013" watchObservedRunningTime="2026-07-17 20:46:03.411300909 +0000 UTC m=+2998.105277056"
Jul 17 20:46:03.421186 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.421161    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"]
Jul 17 20:46:03.428388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.428362    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-snv9n"]
Jul 17 20:46:03.853963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:03.853929    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" path="/var/lib/kubelet/pods/d66a4d85-6b69-4747-9616-7414d9990d82/volumes"
Jul 17 20:46:04.401497 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:04.401466    2572 generic.go:356] "Generic (PLEG): container finished" podID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerID="09b0ae1fb6ad9e13bfbc5531e1d6c58869ed5ebcf1529ab6f3ff754ba94ebcf0" exitCode=6
Jul 17 20:46:04.401670 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:04.401539    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" event={"ID":"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4","Type":"ContainerDied","Data":"09b0ae1fb6ad9e13bfbc5531e1d6c58869ed5ebcf1529ab6f3ff754ba94ebcf0"}
Jul 17 20:46:04.401670 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:04.401593    2572 scope.go:117] "RemoveContainer" containerID="06c9ae4671f80efcc110717e485e0e2d5c20b50f0d3b04686300f68fed6f76a0"
Jul 17 20:46:04.401951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:04.401932    2572 scope.go:117] "RemoveContainer" containerID="09b0ae1fb6ad9e13bfbc5531e1d6c58869ed5ebcf1529ab6f3ff754ba94ebcf0"
Jul 17 20:46:04.402185 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:04.402161    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-ds96l_odh-ai-gateway-infra(5dd4c67f-bfc2-4f7e-a980-ea2d625035e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4"
Jul 17 20:46:04.428416 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:04.428382    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"]
Jul 17 20:46:05.548040 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:05.548013    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"
Jul 17 20:46:05.653607 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:05.653574    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd4c67f-bfc2-4f7e-a980-ea2d625035e4-maas-api-service-ca\") pod \"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4\" (UID: \"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4\") "
Jul 17 20:46:05.653925 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:05.653900    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd4c67f-bfc2-4f7e-a980-ea2d625035e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" (UID: "5dd4c67f-bfc2-4f7e-a980-ea2d625035e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:46:05.754665 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:05.754634    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd4c67f-bfc2-4f7e-a980-ea2d625035e4-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:46:06.414254 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.414226    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"
Jul 17 20:46:06.414440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.414219    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l" event={"ID":"5dd4c67f-bfc2-4f7e-a980-ea2d625035e4","Type":"ContainerDied","Data":"858bcd60d894ede23aaa4de30a5179b056cbdf2240feac15a02a0946fe2e7f23"}
Jul 17 20:46:06.414440 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.414360    2572 scope.go:117] "RemoveContainer" containerID="09b0ae1fb6ad9e13bfbc5531e1d6c58869ed5ebcf1529ab6f3ff754ba94ebcf0"
Jul 17 20:46:06.434336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.434305    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"]
Jul 17 20:46:06.443670 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.443647    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ds96l"]
Jul 17 20:46:06.981095 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981061    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"]
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981379    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981389    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981402    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981407    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981413    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981419    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981427    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981432    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981441    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981446    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981494    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:06.981781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981502    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981507    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:06.981781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981514    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.981781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981519    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:06.981781 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.981527    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" containerName="revoke-keys"
Jul 17 20:46:06.986000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.985980    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:06.988739 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:06.988721    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-327614-serving-cert\""
Jul 17 20:46:07.002585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.002559    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"]
Jul 17 20:46:07.062362 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.062328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.062550 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.062445    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.062550 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.062480    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqzf\" (UniqueName: \"kubernetes.io/projected/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-kube-api-access-drqzf\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.163609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.163576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.163609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.163614    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqzf\" (UniqueName: \"kubernetes.io/projected/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-kube-api-access-drqzf\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.163818 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.163636    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.163818 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:07.163754    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-serving-cert: secret "maas-api-e2e-shared-b-327614-serving-cert" not found
Jul 17 20:46:07.163818 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:07.163807    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls podName:60cfcabc-83c2-42d0-bea0-0ef980bd3c69 nodeName:}" failed. No retries permitted until 2026-07-17 20:46:07.663791403 +0000 UTC m=+3002.357767527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls") pod "maas-api-e2e-shared-b-327614-96565676b-mj2rw" (UID: "60cfcabc-83c2-42d0-bea0-0ef980bd3c69") : secret "maas-api-e2e-shared-b-327614-serving-cert" not found
Jul 17 20:46:07.164311 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.164279    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.179462 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.179427    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqzf\" (UniqueName: \"kubernetes.io/projected/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-kube-api-access-drqzf\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.392111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.392023    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:46:07.668465 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.668365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.671016 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.670990    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls\") pod \"maas-api-e2e-shared-b-327614-96565676b-mj2rw\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:07.853877 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.853845    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd4c67f-bfc2-4f7e-a980-ea2d625035e4" path="/var/lib/kubelet/pods/5dd4c67f-bfc2-4f7e-a980-ea2d625035e4/volumes"
Jul 17 20:46:07.895190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:07.895161    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:08.028935 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:08.028832    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"]
Jul 17 20:46:08.031595 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:46:08.031551    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cfcabc_83c2_42d0_bea0_0ef980bd3c69.slice/crio-981b45460deea2cd654a3e3d715c73c59ed9fb414acd006376653fe17b675d83 WatchSource:0}: Error finding container 981b45460deea2cd654a3e3d715c73c59ed9fb414acd006376653fe17b675d83: Status 404 returned error can't find the container with id 981b45460deea2cd654a3e3d715c73c59ed9fb414acd006376653fe17b675d83
Jul 17 20:46:08.426372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:08.426319    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw" event={"ID":"60cfcabc-83c2-42d0-bea0-0ef980bd3c69","Type":"ContainerStarted","Data":"981b45460deea2cd654a3e3d715c73c59ed9fb414acd006376653fe17b675d83"}
Jul 17 20:46:08.848069 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:08.848013    2572 scope.go:117] "RemoveContainer" containerID="542b663a9ec56da3ea5d0be5513adb6876af68e35fb945ac2cc5464e660eb8b4"
Jul 17 20:46:09.435322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:09.435227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw" event={"ID":"60cfcabc-83c2-42d0-bea0-0ef980bd3c69","Type":"ContainerStarted","Data":"a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc"}
Jul 17 20:46:09.435742 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:09.435395    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:09.436979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:09.436957    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" event={"ID":"1301b6d6-5c62-4e93-ab91-386ba2e7378f","Type":"ContainerStarted","Data":"cffe9e9dfbe4835101a4ea29c811eddc9567f34b2593f9ce9042ff0a1aba5407"}
Jul 17 20:46:09.453551 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:09.453504    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw" podStartSLOduration=3.013869034 podStartE2EDuration="3.453489712s" totalImagesPullingTime="439.620678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:09.451802063 +0000 UTC m=+3004.145778208" watchObservedRunningTime="2026-07-17 20:46:09.453489712 +0000 UTC m=+3004.147465857"
Jul 17 20:46:09.470172 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:09.470131    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" podStartSLOduration=20.470118326 podStartE2EDuration="20.470118326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:09.468752308 +0000 UTC m=+3004.162728448" watchObservedRunningTime="2026-07-17 20:46:09.470118326 +0000 UTC m=+3004.164094472"
Jul 17 20:46:10.444791 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:10.444759    2572 generic.go:356] "Generic (PLEG): container finished" podID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerID="cffe9e9dfbe4835101a4ea29c811eddc9567f34b2593f9ce9042ff0a1aba5407" exitCode=6
Jul 17 20:46:10.445230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:10.444828    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" event={"ID":"1301b6d6-5c62-4e93-ab91-386ba2e7378f","Type":"ContainerDied","Data":"cffe9e9dfbe4835101a4ea29c811eddc9567f34b2593f9ce9042ff0a1aba5407"}
Jul 17 20:46:10.445230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:10.444883    2572 scope.go:117] "RemoveContainer" containerID="542b663a9ec56da3ea5d0be5513adb6876af68e35fb945ac2cc5464e660eb8b4"
Jul 17 20:46:10.445230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:10.445199    2572 scope.go:117] "RemoveContainer" containerID="cffe9e9dfbe4835101a4ea29c811eddc9567f34b2593f9ce9042ff0a1aba5407"
Jul 17 20:46:10.445379 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:10.445362    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9_odh-ai-gateway-infra(1301b6d6-5c62-4e93-ab91-386ba2e7378f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f"
Jul 17 20:46:10.479368 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:10.479343    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"]
Jul 17 20:46:11.583178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:11.583152    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"
Jul 17 20:46:11.703784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:11.703687    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1301b6d6-5c62-4e93-ab91-386ba2e7378f-maas-api-service-ca\") pod \"1301b6d6-5c62-4e93-ab91-386ba2e7378f\" (UID: \"1301b6d6-5c62-4e93-ab91-386ba2e7378f\") "
Jul 17 20:46:11.704128 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:11.704106    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1301b6d6-5c62-4e93-ab91-386ba2e7378f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1301b6d6-5c62-4e93-ab91-386ba2e7378f" (UID: "1301b6d6-5c62-4e93-ab91-386ba2e7378f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:46:11.804775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:11.804738    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1301b6d6-5c62-4e93-ab91-386ba2e7378f-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:46:12.456097 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:12.456071    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"
Jul 17 20:46:12.456299 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:12.456067    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9" event={"ID":"1301b6d6-5c62-4e93-ab91-386ba2e7378f","Type":"ContainerDied","Data":"82b37da6e358355c2a06e628497071cd5ec2fa35caa9cda8fd46a712f830aef0"}
Jul 17 20:46:12.456299 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:12.456198    2572 scope.go:117] "RemoveContainer" containerID="cffe9e9dfbe4835101a4ea29c811eddc9567f34b2593f9ce9042ff0a1aba5407"
Jul 17 20:46:12.477239 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:12.477210    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"]
Jul 17 20:46:12.484169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:12.484145    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-589n9"]
Jul 17 20:46:13.852836 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:13.852803    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" path="/var/lib/kubelet/pods/1301b6d6-5c62-4e93-ab91-386ba2e7378f/volumes"
Jul 17 20:46:14.389144 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:14.389071    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:46:15.450388 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:15.450363    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:46:23.897631 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:23.897592    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:46:35.691566 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691526    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"]
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691867    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691880    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691892    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691900    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691920    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691925    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66a4d85-6b69-4747-9616-7414d9990d82" containerName="revoke-keys"
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691972    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:35.692042 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.691984    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:35.697523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.697496    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"
Jul 17 20:46:35.700743 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.700719    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:46:35.702218 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.702190    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"]
Jul 17 20:46:35.805200 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.805166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8945f158-b0a1-4ae9-94da-33a64f920ec8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz\" (UID: \"8945f158-b0a1-4ae9-94da-33a64f920ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"
Jul 17 20:46:35.906540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.906506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8945f158-b0a1-4ae9-94da-33a64f920ec8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz\" (UID: \"8945f158-b0a1-4ae9-94da-33a64f920ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"
Jul 17 20:46:35.907174 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:35.907153    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8945f158-b0a1-4ae9-94da-33a64f920ec8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz\" (UID: \"8945f158-b0a1-4ae9-94da-33a64f920ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"
Jul 17 20:46:36.009768 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:36.009735    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"
Jul 17 20:46:36.143807 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:36.143783    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"]
Jul 17 20:46:36.145534 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:46:36.145505    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8945f158_b0a1_4ae9_94da_33a64f920ec8.slice/crio-1fc55575fb96a6e91c82b95f179c295bbc4893964e19620620c6f1b4357776ca WatchSource:0}: Error finding container 1fc55575fb96a6e91c82b95f179c295bbc4893964e19620620c6f1b4357776ca: Status 404 returned error can't find the container with id 1fc55575fb96a6e91c82b95f179c295bbc4893964e19620620c6f1b4357776ca
Jul 17 20:46:36.567342 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:36.567308    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" event={"ID":"8945f158-b0a1-4ae9-94da-33a64f920ec8","Type":"ContainerStarted","Data":"e79033a72075e0103b4050172a3af00abbbb21c1d779aea78bbe51fabcc1572a"}
Jul 17 20:46:36.567504 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:36.567358    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" event={"ID":"8945f158-b0a1-4ae9-94da-33a64f920ec8","Type":"ContainerStarted","Data":"1fc55575fb96a6e91c82b95f179c295bbc4893964e19620620c6f1b4357776ca"}
Jul 17 20:46:36.584241 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:36.584143    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" podStartSLOduration=1.584129074 podStartE2EDuration="1.584129074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:36.581673413 +0000 UTC m=+3031.275649558" watchObservedRunningTime="2026-07-17 20:46:36.584129074 +0000 UTC m=+3031.278105219"
Jul 17 20:46:37.573766 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.573725    2572 generic.go:356] "Generic (PLEG): container finished" podID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerID="e79033a72075e0103b4050172a3af00abbbb21c1d779aea78bbe51fabcc1572a" exitCode=6
Jul 17 20:46:37.573766 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.573762    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" event={"ID":"8945f158-b0a1-4ae9-94da-33a64f920ec8","Type":"ContainerDied","Data":"e79033a72075e0103b4050172a3af00abbbb21c1d779aea78bbe51fabcc1572a"}
Jul 17 20:46:37.574375 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.574026    2572 scope.go:117] "RemoveContainer" containerID="e79033a72075e0103b4050172a3af00abbbb21c1d779aea78bbe51fabcc1572a"
Jul 17 20:46:37.895032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.895000    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"]
Jul 17 20:46:37.895370 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.895357    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:37.895416 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.895372    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:37.895456 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.895434    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1301b6d6-5c62-4e93-ab91-386ba2e7378f" containerName="revoke-keys"
Jul 17 20:46:37.898328 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.898313    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"
Jul 17 20:46:37.905626 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:37.905316    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"]
Jul 17 20:46:38.027584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.027552    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2396b0-965c-4b2b-93b0-35e0b94e4c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr\" (UID: \"5d2396b0-965c-4b2b-93b0-35e0b94e4c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"
Jul 17 20:46:38.128446 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.128421    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2396b0-965c-4b2b-93b0-35e0b94e4c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr\" (UID: \"5d2396b0-965c-4b2b-93b0-35e0b94e4c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"
Jul 17 20:46:38.129084 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.129031    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2396b0-965c-4b2b-93b0-35e0b94e4c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr\" (UID: \"5d2396b0-965c-4b2b-93b0-35e0b94e4c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"
Jul 17 20:46:38.209462 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.209429    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"
Jul 17 20:46:38.338668 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.338642    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"]
Jul 17 20:46:38.340745 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:46:38.340722    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2396b0_965c_4b2b_93b0_35e0b94e4c21.slice/crio-cbf977c859effd643accaa423a7bc43ca075e2dd3fb2cab292d682a3f13ba712 WatchSource:0}: Error finding container cbf977c859effd643accaa423a7bc43ca075e2dd3fb2cab292d682a3f13ba712: Status 404 returned error can't find the container with id cbf977c859effd643accaa423a7bc43ca075e2dd3fb2cab292d682a3f13ba712
Jul 17 20:46:38.582451 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.582421    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" event={"ID":"5d2396b0-965c-4b2b-93b0-35e0b94e4c21","Type":"ContainerStarted","Data":"4b8c99d339a0a4092b580dbc3cac1427e52ddb59a964887770b55cad703d4ae8"}
Jul 17 20:46:38.582904 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.582475    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" event={"ID":"5d2396b0-965c-4b2b-93b0-35e0b94e4c21","Type":"ContainerStarted","Data":"cbf977c859effd643accaa423a7bc43ca075e2dd3fb2cab292d682a3f13ba712"}
Jul 17 20:46:38.584128 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.584102    2572 generic.go:356] "Generic (PLEG): container finished" podID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerID="cb5891ee1e2cc8703cd09a9c03d524f6e6035378d87709fb4ffb49e9361e8551" exitCode=6
Jul 17 20:46:38.584228 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.584197    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" event={"ID":"8945f158-b0a1-4ae9-94da-33a64f920ec8","Type":"ContainerDied","Data":"cb5891ee1e2cc8703cd09a9c03d524f6e6035378d87709fb4ffb49e9361e8551"}
Jul 17 20:46:38.584274 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.584263    2572 scope.go:117] "RemoveContainer" containerID="e79033a72075e0103b4050172a3af00abbbb21c1d779aea78bbe51fabcc1572a"
Jul 17 20:46:38.584370 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.584357    2572 scope.go:117] "RemoveContainer" containerID="cb5891ee1e2cc8703cd09a9c03d524f6e6035378d87709fb4ffb49e9361e8551"
Jul 17 20:46:38.584546 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:38.584526    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz_odh-ai-gateway-infra(8945f158-b0a1-4ae9-94da-33a64f920ec8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8"
Jul 17 20:46:38.606762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:38.606718    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" podStartSLOduration=1.606701532 podStartE2EDuration="1.606701532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:38.606517086 +0000 UTC m=+3033.300493238" watchObservedRunningTime="2026-07-17 20:46:38.606701532 +0000 UTC m=+3033.300677656"
Jul 17 20:46:39.590999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:39.590970    2572 scope.go:117] "RemoveContainer" containerID="cb5891ee1e2cc8703cd09a9c03d524f6e6035378d87709fb4ffb49e9361e8551"
Jul 17 20:46:39.591406 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:39.591180    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz_odh-ai-gateway-infra(8945f158-b0a1-4ae9-94da-33a64f920ec8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8"
Jul 17 20:46:39.592333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:39.592313    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerID="4b8c99d339a0a4092b580dbc3cac1427e52ddb59a964887770b55cad703d4ae8" exitCode=6
Jul 17 20:46:39.592429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:39.592348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" event={"ID":"5d2396b0-965c-4b2b-93b0-35e0b94e4c21","Type":"ContainerDied","Data":"4b8c99d339a0a4092b580dbc3cac1427e52ddb59a964887770b55cad703d4ae8"}
Jul 17 20:46:39.592524 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:39.592513    2572 scope.go:117] "RemoveContainer" containerID="4b8c99d339a0a4092b580dbc3cac1427e52ddb59a964887770b55cad703d4ae8"
Jul 17 20:46:40.598727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:40.598631    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerID="8d3484436689d45e3d5866924b25470037a49f1bcd9cbc39aab7cda509951b5f" exitCode=6
Jul 17 20:46:40.599180 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:40.598726    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" event={"ID":"5d2396b0-965c-4b2b-93b0-35e0b94e4c21","Type":"ContainerDied","Data":"8d3484436689d45e3d5866924b25470037a49f1bcd9cbc39aab7cda509951b5f"}
Jul 17 20:46:40.599180 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:40.598795    2572 scope.go:117] "RemoveContainer" containerID="4b8c99d339a0a4092b580dbc3cac1427e52ddb59a964887770b55cad703d4ae8"
Jul 17 20:46:40.599180 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:40.598912    2572 scope.go:117] "RemoveContainer" containerID="8d3484436689d45e3d5866924b25470037a49f1bcd9cbc39aab7cda509951b5f"
Jul 17 20:46:40.599180 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:40.599148    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr_odh-ai-gateway-infra(5d2396b0-965c-4b2b-93b0-35e0b94e4c21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21"
Jul 17 20:46:41.606411 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:41.606381    2572 scope.go:117] "RemoveContainer" containerID="8d3484436689d45e3d5866924b25470037a49f1bcd9cbc39aab7cda509951b5f"
Jul 17 20:46:41.606807 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:41.606570    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr_odh-ai-gateway-infra(5d2396b0-965c-4b2b-93b0-35e0b94e4c21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21"
Jul 17 20:46:44.866219 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:44.866187    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:46:49.705534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:49.705495    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"]
Jul 17 20:46:49.708757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:49.708741    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"
Jul 17 20:46:49.712561 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:49.712536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2af873-4cc9-44e7-9901-c3e60598a090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq\" (UID: \"3f2af873-4cc9-44e7-9901-c3e60598a090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"
Jul 17 20:46:49.717260 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:49.717231    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"]
Jul 17 20:46:49.814003 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:49.813958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2af873-4cc9-44e7-9901-c3e60598a090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq\" (UID: \"3f2af873-4cc9-44e7-9901-c3e60598a090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"
Jul 17 20:46:49.814642 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:49.814620    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2af873-4cc9-44e7-9901-c3e60598a090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq\" (UID: \"3f2af873-4cc9-44e7-9901-c3e60598a090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"
Jul 17 20:46:49.848449 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:49.848419    2572 scope.go:117] "RemoveContainer" containerID="cb5891ee1e2cc8703cd09a9c03d524f6e6035378d87709fb4ffb49e9361e8551"
Jul 17 20:46:50.020185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.020161    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"
Jul 17 20:46:50.157405 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.157376    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"]
Jul 17 20:46:50.160661 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:46:50.160638    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2af873_4cc9_44e7_9901_c3e60598a090.slice/crio-ed5d0df328fe5051df28d1934c257fd235edb68349cedb6db2b052d71d214233 WatchSource:0}: Error finding container ed5d0df328fe5051df28d1934c257fd235edb68349cedb6db2b052d71d214233: Status 404 returned error can't find the container with id ed5d0df328fe5051df28d1934c257fd235edb68349cedb6db2b052d71d214233
Jul 17 20:46:50.653282 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.653256    2572 generic.go:356] "Generic (PLEG): container finished" podID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerID="7ee6875bd5bef62be07e36faff8ead26cc22d14ba09b42e46a69ab29f9649aa9" exitCode=6
Jul 17 20:46:50.653420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.653326    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" event={"ID":"8945f158-b0a1-4ae9-94da-33a64f920ec8","Type":"ContainerDied","Data":"7ee6875bd5bef62be07e36faff8ead26cc22d14ba09b42e46a69ab29f9649aa9"}
Jul 17 20:46:50.653420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.653404    2572 scope.go:117] "RemoveContainer" containerID="cb5891ee1e2cc8703cd09a9c03d524f6e6035378d87709fb4ffb49e9361e8551"
Jul 17 20:46:50.653761 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.653742    2572 scope.go:117] "RemoveContainer" containerID="7ee6875bd5bef62be07e36faff8ead26cc22d14ba09b42e46a69ab29f9649aa9"
Jul 17 20:46:50.653951 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:50.653934    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz_odh-ai-gateway-infra(8945f158-b0a1-4ae9-94da-33a64f920ec8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8"
Jul 17 20:46:50.654789 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.654769    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" event={"ID":"3f2af873-4cc9-44e7-9901-c3e60598a090","Type":"ContainerStarted","Data":"e4ea9339b2f24d0262a15d751de6df15d1621dee4846c0dc27df0d4b03bbd11b"}
Jul 17 20:46:50.654923 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.654809    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" event={"ID":"3f2af873-4cc9-44e7-9901-c3e60598a090","Type":"ContainerStarted","Data":"ed5d0df328fe5051df28d1934c257fd235edb68349cedb6db2b052d71d214233"}
Jul 17 20:46:50.682891 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:50.682844    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" podStartSLOduration=1.682828433 podStartE2EDuration="1.682828433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:46:50.681353025 +0000 UTC m=+3045.375329164" watchObservedRunningTime="2026-07-17 20:46:50.682828433 +0000 UTC m=+3045.376804636"
Jul 17 20:46:51.663038 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:51.663003    2572 generic.go:356] "Generic (PLEG): container finished" podID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerID="e4ea9339b2f24d0262a15d751de6df15d1621dee4846c0dc27df0d4b03bbd11b" exitCode=6
Jul 17 20:46:51.663496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:51.663091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" event={"ID":"3f2af873-4cc9-44e7-9901-c3e60598a090","Type":"ContainerDied","Data":"e4ea9339b2f24d0262a15d751de6df15d1621dee4846c0dc27df0d4b03bbd11b"}
Jul 17 20:46:51.663496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:51.663265    2572 scope.go:117] "RemoveContainer" containerID="e4ea9339b2f24d0262a15d751de6df15d1621dee4846c0dc27df0d4b03bbd11b"
Jul 17 20:46:51.677963 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:51.677939    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"]
Jul 17 20:46:51.815083 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:51.815059    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"
Jul 17 20:46:51.926334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:51.926309    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8945f158-b0a1-4ae9-94da-33a64f920ec8-maas-api-service-ca\") pod \"8945f158-b0a1-4ae9-94da-33a64f920ec8\" (UID: \"8945f158-b0a1-4ae9-94da-33a64f920ec8\") "
Jul 17 20:46:51.926686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:51.926665    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8945f158-b0a1-4ae9-94da-33a64f920ec8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8945f158-b0a1-4ae9-94da-33a64f920ec8" (UID: "8945f158-b0a1-4ae9-94da-33a64f920ec8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:46:52.027420 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.027376    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8945f158-b0a1-4ae9-94da-33a64f920ec8-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:46:52.669220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.669136    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"
Jul 17 20:46:52.669220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.669145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz" event={"ID":"8945f158-b0a1-4ae9-94da-33a64f920ec8","Type":"ContainerDied","Data":"1fc55575fb96a6e91c82b95f179c295bbc4893964e19620620c6f1b4357776ca"}
Jul 17 20:46:52.669220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.669210    2572 scope.go:117] "RemoveContainer" containerID="7ee6875bd5bef62be07e36faff8ead26cc22d14ba09b42e46a69ab29f9649aa9"
Jul 17 20:46:52.671446 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.671407    2572 generic.go:356] "Generic (PLEG): container finished" podID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerID="a89fa052273d46d3c522a1191edcdd09f088db1172bc5efbe4bb77f706e4d6ec" exitCode=6
Jul 17 20:46:52.671582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.671457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" event={"ID":"3f2af873-4cc9-44e7-9901-c3e60598a090","Type":"ContainerDied","Data":"a89fa052273d46d3c522a1191edcdd09f088db1172bc5efbe4bb77f706e4d6ec"}
Jul 17 20:46:52.671706 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.671690    2572 scope.go:117] "RemoveContainer" containerID="a89fa052273d46d3c522a1191edcdd09f088db1172bc5efbe4bb77f706e4d6ec"
Jul 17 20:46:52.671904 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:52.671886    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq_odh-ai-gateway-infra(3f2af873-4cc9-44e7-9901-c3e60598a090)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090"
Jul 17 20:46:52.678430 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.678414    2572 scope.go:117] "RemoveContainer" containerID="e4ea9339b2f24d0262a15d751de6df15d1621dee4846c0dc27df0d4b03bbd11b"
Jul 17 20:46:52.701423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.701388    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"]
Jul 17 20:46:52.705127 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:52.705108    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8s7tz"]
Jul 17 20:46:53.680230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:53.680200    2572 scope.go:117] "RemoveContainer" containerID="a89fa052273d46d3c522a1191edcdd09f088db1172bc5efbe4bb77f706e4d6ec"
Jul 17 20:46:53.680630 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:53.680389    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq_odh-ai-gateway-infra(3f2af873-4cc9-44e7-9901-c3e60598a090)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090"
Jul 17 20:46:53.848529 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:53.848501    2572 scope.go:117] "RemoveContainer" containerID="8d3484436689d45e3d5866924b25470037a49f1bcd9cbc39aab7cda509951b5f"
Jul 17 20:46:53.854348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:53.854323    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" path="/var/lib/kubelet/pods/8945f158-b0a1-4ae9-94da-33a64f920ec8/volumes"
Jul 17 20:46:54.686761 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:54.686731    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerID="846fc3a99c8bc16010b8b90f77fae5273afefdd2e43a49059599d0fad2277f05" exitCode=6
Jul 17 20:46:54.687216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:54.686797    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" event={"ID":"5d2396b0-965c-4b2b-93b0-35e0b94e4c21","Type":"ContainerDied","Data":"846fc3a99c8bc16010b8b90f77fae5273afefdd2e43a49059599d0fad2277f05"}
Jul 17 20:46:54.687216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:54.686859    2572 scope.go:117] "RemoveContainer" containerID="8d3484436689d45e3d5866924b25470037a49f1bcd9cbc39aab7cda509951b5f"
Jul 17 20:46:54.687216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:54.687189    2572 scope.go:117] "RemoveContainer" containerID="846fc3a99c8bc16010b8b90f77fae5273afefdd2e43a49059599d0fad2277f05"
Jul 17 20:46:54.687379 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:46:54.687360    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr_odh-ai-gateway-infra(5d2396b0-965c-4b2b-93b0-35e0b94e4c21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21"
Jul 17 20:46:55.713436 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:55.713388    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"]
Jul 17 20:46:55.849782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:55.849751    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"
Jul 17 20:46:55.856368 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:55.856346    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2396b0-965c-4b2b-93b0-35e0b94e4c21-maas-api-service-ca\") pod \"5d2396b0-965c-4b2b-93b0-35e0b94e4c21\" (UID: \"5d2396b0-965c-4b2b-93b0-35e0b94e4c21\") "
Jul 17 20:46:55.856655 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:55.856634    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2396b0-965c-4b2b-93b0-35e0b94e4c21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d2396b0-965c-4b2b-93b0-35e0b94e4c21" (UID: "5d2396b0-965c-4b2b-93b0-35e0b94e4c21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:46:55.957674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:55.957635    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2396b0-965c-4b2b-93b0-35e0b94e4c21-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:46:56.701303 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:56.701267    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"
Jul 17 20:46:56.701484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:56.701269    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr" event={"ID":"5d2396b0-965c-4b2b-93b0-35e0b94e4c21","Type":"ContainerDied","Data":"cbf977c859effd643accaa423a7bc43ca075e2dd3fb2cab292d682a3f13ba712"}
Jul 17 20:46:56.701484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:56.701404    2572 scope.go:117] "RemoveContainer" containerID="846fc3a99c8bc16010b8b90f77fae5273afefdd2e43a49059599d0fad2277f05"
Jul 17 20:46:56.724971 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:56.724944    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"]
Jul 17 20:46:56.733276 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:56.733244    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-n9rlr"]
Jul 17 20:46:57.853593 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:46:57.853518    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" path="/var/lib/kubelet/pods/5d2396b0-965c-4b2b-93b0-35e0b94e4c21/volumes"
Jul 17 20:47:04.848070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:04.848008    2572 scope.go:117] "RemoveContainer" containerID="a89fa052273d46d3c522a1191edcdd09f088db1172bc5efbe4bb77f706e4d6ec"
Jul 17 20:47:05.746357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:05.746324    2572 generic.go:356] "Generic (PLEG): container finished" podID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerID="ceda93d3dca2db4ce7e4603709b119bd564e9daf9eb65a5332646517ff79789d" exitCode=6
Jul 17 20:47:05.746588 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:05.746401    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" event={"ID":"3f2af873-4cc9-44e7-9901-c3e60598a090","Type":"ContainerDied","Data":"ceda93d3dca2db4ce7e4603709b119bd564e9daf9eb65a5332646517ff79789d"}
Jul 17 20:47:05.746588 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:05.746476    2572 scope.go:117] "RemoveContainer" containerID="a89fa052273d46d3c522a1191edcdd09f088db1172bc5efbe4bb77f706e4d6ec"
Jul 17 20:47:05.746843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:05.746807    2572 scope.go:117] "RemoveContainer" containerID="ceda93d3dca2db4ce7e4603709b119bd564e9daf9eb65a5332646517ff79789d"
Jul 17 20:47:05.747034 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:05.747013    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq_odh-ai-gateway-infra(3f2af873-4cc9-44e7-9901-c3e60598a090)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090"
Jul 17 20:47:06.773250 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:06.773216    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"]
Jul 17 20:47:06.907655 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:06.907624    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"
Jul 17 20:47:06.939648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:06.939619    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2af873-4cc9-44e7-9901-c3e60598a090-maas-api-service-ca\") pod \"3f2af873-4cc9-44e7-9901-c3e60598a090\" (UID: \"3f2af873-4cc9-44e7-9901-c3e60598a090\") "
Jul 17 20:47:06.939989 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:06.939952    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2af873-4cc9-44e7-9901-c3e60598a090-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f2af873-4cc9-44e7-9901-c3e60598a090" (UID: "3f2af873-4cc9-44e7-9901-c3e60598a090"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:47:07.040647 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.040565    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2af873-4cc9-44e7-9901-c3e60598a090-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:47:07.765799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.765768    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"
Jul 17 20:47:07.765995 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.765804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq" event={"ID":"3f2af873-4cc9-44e7-9901-c3e60598a090","Type":"ContainerDied","Data":"ed5d0df328fe5051df28d1934c257fd235edb68349cedb6db2b052d71d214233"}
Jul 17 20:47:07.765995 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.765864    2572 scope.go:117] "RemoveContainer" containerID="ceda93d3dca2db4ce7e4603709b119bd564e9daf9eb65a5332646517ff79789d"
Jul 17 20:47:07.790578 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.790554    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"]
Jul 17 20:47:07.796742 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.796720    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dlmtq"]
Jul 17 20:47:07.853715 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.853678    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" path="/var/lib/kubelet/pods/3f2af873-4cc9-44e7-9901-c3e60598a090/volumes"
Jul 17 20:47:07.980426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:07.980390    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:47:11.738943 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.738905    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"]
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739312    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739325    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739333    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739338    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739347    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739353    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739362    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739367    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.739372 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739373    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739379    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739386    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739391    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739398    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739403    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739414    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739420    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739468    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739476    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739482    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739487    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739493    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739498    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:11.739686 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.739506    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:11.745514 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.745498    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"
Jul 17 20:47:11.748728 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.748698    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:47:11.750332 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.750311    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"]
Jul 17 20:47:11.775466 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.775435    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039d2ff8-52e4-4560-8de5-cb8d0826718b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-9x88t\" (UID: \"039d2ff8-52e4-4560-8de5-cb8d0826718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"
Jul 17 20:47:11.876356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.876319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039d2ff8-52e4-4560-8de5-cb8d0826718b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-9x88t\" (UID: \"039d2ff8-52e4-4560-8de5-cb8d0826718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"
Jul 17 20:47:11.876990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:11.876971    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039d2ff8-52e4-4560-8de5-cb8d0826718b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-9x88t\" (UID: \"039d2ff8-52e4-4560-8de5-cb8d0826718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"
Jul 17 20:47:12.058016 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:12.057933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"
Jul 17 20:47:12.203562 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:12.203538    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"]
Jul 17 20:47:12.206128 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:47:12.206097    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039d2ff8_52e4_4560_8de5_cb8d0826718b.slice/crio-f6a3f69d4a2dc61a2da3631a26b0ef3353acbd9ccba2fdd01d2fefb554638504 WatchSource:0}: Error finding container f6a3f69d4a2dc61a2da3631a26b0ef3353acbd9ccba2fdd01d2fefb554638504: Status 404 returned error can't find the container with id f6a3f69d4a2dc61a2da3631a26b0ef3353acbd9ccba2fdd01d2fefb554638504
Jul 17 20:47:12.790623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:12.790585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" event={"ID":"039d2ff8-52e4-4560-8de5-cb8d0826718b","Type":"ContainerStarted","Data":"28b42c20796aba6021bd0126d102d0f0528c6e997894522c2526b10db5e8617f"}
Jul 17 20:47:12.791017 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:12.790674    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" event={"ID":"039d2ff8-52e4-4560-8de5-cb8d0826718b","Type":"ContainerStarted","Data":"f6a3f69d4a2dc61a2da3631a26b0ef3353acbd9ccba2fdd01d2fefb554638504"}
Jul 17 20:47:12.809636 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:12.809583    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" podStartSLOduration=1.809568509 podStartE2EDuration="1.809568509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:47:12.80775961 +0000 UTC m=+3067.501735769" watchObservedRunningTime="2026-07-17 20:47:12.809568509 +0000 UTC m=+3067.503544655"
Jul 17 20:47:13.797096 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:13.797040    2572 generic.go:356] "Generic (PLEG): container finished" podID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerID="28b42c20796aba6021bd0126d102d0f0528c6e997894522c2526b10db5e8617f" exitCode=6
Jul 17 20:47:13.797546 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:13.797134    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" event={"ID":"039d2ff8-52e4-4560-8de5-cb8d0826718b","Type":"ContainerDied","Data":"28b42c20796aba6021bd0126d102d0f0528c6e997894522c2526b10db5e8617f"}
Jul 17 20:47:13.797546 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:13.797308    2572 scope.go:117] "RemoveContainer" containerID="28b42c20796aba6021bd0126d102d0f0528c6e997894522c2526b10db5e8617f"
Jul 17 20:47:14.804024 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:14.803932    2572 generic.go:356] "Generic (PLEG): container finished" podID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerID="15a6541959d3faeb4c0b2ba3dfe069573f827cfab8d3e63051a86bf36ae0a35e" exitCode=6
Jul 17 20:47:14.804024 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:14.804011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" event={"ID":"039d2ff8-52e4-4560-8de5-cb8d0826718b","Type":"ContainerDied","Data":"15a6541959d3faeb4c0b2ba3dfe069573f827cfab8d3e63051a86bf36ae0a35e"}
Jul 17 20:47:14.804552 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:14.804092    2572 scope.go:117] "RemoveContainer" containerID="28b42c20796aba6021bd0126d102d0f0528c6e997894522c2526b10db5e8617f"
Jul 17 20:47:14.804552 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:14.804332    2572 scope.go:117] "RemoveContainer" containerID="15a6541959d3faeb4c0b2ba3dfe069573f827cfab8d3e63051a86bf36ae0a35e"
Jul 17 20:47:14.804552 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:14.804525    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-9x88t_odh-ai-gateway-infra(039d2ff8-52e4-4560-8de5-cb8d0826718b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b"
Jul 17 20:47:15.809680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:15.809651    2572 scope.go:117] "RemoveContainer" containerID="15a6541959d3faeb4c0b2ba3dfe069573f827cfab8d3e63051a86bf36ae0a35e"
Jul 17 20:47:15.810077 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:15.809845    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-9x88t_odh-ai-gateway-infra(039d2ff8-52e4-4560-8de5-cb8d0826718b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b"
Jul 17 20:47:25.740009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.739966    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"]
Jul 17 20:47:25.740525 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.740488    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:25.740525 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.740507    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2af873-4cc9-44e7-9901-c3e60598a090" containerName="revoke-keys"
Jul 17 20:47:25.740656 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.740617    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2396b0-965c-4b2b-93b0-35e0b94e4c21" containerName="revoke-keys"
Jul 17 20:47:25.740656 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.740635    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8945f158-b0a1-4ae9-94da-33a64f920ec8" containerName="revoke-keys"
Jul 17 20:47:25.743712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.743692    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"
Jul 17 20:47:25.751892 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.751845    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"]
Jul 17 20:47:25.794443 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.794396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80d0b6a-4df8-49a1-8f6f-887707b3a9ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s\" (UID: \"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"
Jul 17 20:47:25.895197 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.895153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80d0b6a-4df8-49a1-8f6f-887707b3a9ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s\" (UID: \"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"
Jul 17 20:47:25.895874 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:25.895842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80d0b6a-4df8-49a1-8f6f-887707b3a9ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s\" (UID: \"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"
Jul 17 20:47:26.062470 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:26.062371    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"
Jul 17 20:47:26.400169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:26.400017    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"]
Jul 17 20:47:26.402457 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:47:26.402430    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80d0b6a_4df8_49a1_8f6f_887707b3a9ba.slice/crio-60a1317b11d5e814da6cfdff0d19926f583fa457e3fd45912b1b5b6021d2aad2 WatchSource:0}: Error finding container 60a1317b11d5e814da6cfdff0d19926f583fa457e3fd45912b1b5b6021d2aad2: Status 404 returned error can't find the container with id 60a1317b11d5e814da6cfdff0d19926f583fa457e3fd45912b1b5b6021d2aad2
Jul 17 20:47:26.480228 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:26.480189    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:47:26.863836 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:26.863796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" event={"ID":"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba","Type":"ContainerStarted","Data":"d020029a9e88503edbad280b18d8ebabe03ea3dfcfd69e3bb7ac03b41c60517f"}
Jul 17 20:47:26.864225 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:26.863857    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" event={"ID":"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba","Type":"ContainerStarted","Data":"60a1317b11d5e814da6cfdff0d19926f583fa457e3fd45912b1b5b6021d2aad2"}
Jul 17 20:47:26.881079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:26.881017    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" podStartSLOduration=1.881000182 podStartE2EDuration="1.881000182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:47:26.879395862 +0000 UTC m=+3081.573372033" watchObservedRunningTime="2026-07-17 20:47:26.881000182 +0000 UTC m=+3081.574976348"
Jul 17 20:47:27.848258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:27.848225    2572 scope.go:117] "RemoveContainer" containerID="15a6541959d3faeb4c0b2ba3dfe069573f827cfab8d3e63051a86bf36ae0a35e"
Jul 17 20:47:27.870149 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:27.870122    2572 generic.go:356] "Generic (PLEG): container finished" podID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerID="d020029a9e88503edbad280b18d8ebabe03ea3dfcfd69e3bb7ac03b41c60517f" exitCode=6
Jul 17 20:47:27.870479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:27.870206    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" event={"ID":"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba","Type":"ContainerDied","Data":"d020029a9e88503edbad280b18d8ebabe03ea3dfcfd69e3bb7ac03b41c60517f"}
Jul 17 20:47:27.870479 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:27.870337    2572 scope.go:117] "RemoveContainer" containerID="d020029a9e88503edbad280b18d8ebabe03ea3dfcfd69e3bb7ac03b41c60517f"
Jul 17 20:47:27.940008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:27.939975    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"]
Jul 17 20:47:27.944528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:27.944511    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"
Jul 17 20:47:27.949935 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:27.949910    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"]
Jul 17 20:47:28.011387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.011339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abff9fe3-c452-41f5-a010-846a0abf05ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl\" (UID: \"abff9fe3-c452-41f5-a010-846a0abf05ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"
Jul 17 20:47:28.111915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.111812    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abff9fe3-c452-41f5-a010-846a0abf05ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl\" (UID: \"abff9fe3-c452-41f5-a010-846a0abf05ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"
Jul 17 20:47:28.112522 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.112499    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abff9fe3-c452-41f5-a010-846a0abf05ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl\" (UID: \"abff9fe3-c452-41f5-a010-846a0abf05ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"
Jul 17 20:47:28.256639 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.256611    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"
Jul 17 20:47:28.384627 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.384604    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"]
Jul 17 20:47:28.386773 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:47:28.386735    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabff9fe3_c452_41f5_a010_846a0abf05ba.slice/crio-ffa236f826e52692ffc96ad9d77d65b3c77f73ffb938f6b7f59c1cc62d2e28da WatchSource:0}: Error finding container ffa236f826e52692ffc96ad9d77d65b3c77f73ffb938f6b7f59c1cc62d2e28da: Status 404 returned error can't find the container with id ffa236f826e52692ffc96ad9d77d65b3c77f73ffb938f6b7f59c1cc62d2e28da
Jul 17 20:47:28.877031 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.876998    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" event={"ID":"abff9fe3-c452-41f5-a010-846a0abf05ba","Type":"ContainerStarted","Data":"b541c3715855ceb400fd009dd74db6cc1d564ea63ab1445a23be2234a72ab53f"}
Jul 17 20:47:28.877477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.877080    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" event={"ID":"abff9fe3-c452-41f5-a010-846a0abf05ba","Type":"ContainerStarted","Data":"ffa236f826e52692ffc96ad9d77d65b3c77f73ffb938f6b7f59c1cc62d2e28da"}
Jul 17 20:47:28.878887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.878863    2572 generic.go:356] "Generic (PLEG): container finished" podID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerID="4a85f0ab714294390bb936118a54483a96f185b74aa32b7d1348e8d369041c70" exitCode=6
Jul 17 20:47:28.878953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.878930    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" event={"ID":"039d2ff8-52e4-4560-8de5-cb8d0826718b","Type":"ContainerDied","Data":"4a85f0ab714294390bb936118a54483a96f185b74aa32b7d1348e8d369041c70"}
Jul 17 20:47:28.879004 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.878976    2572 scope.go:117] "RemoveContainer" containerID="15a6541959d3faeb4c0b2ba3dfe069573f827cfab8d3e63051a86bf36ae0a35e"
Jul 17 20:47:28.879300 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.879285    2572 scope.go:117] "RemoveContainer" containerID="4a85f0ab714294390bb936118a54483a96f185b74aa32b7d1348e8d369041c70"
Jul 17 20:47:28.879507 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:28.879485    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-9x88t_odh-ai-gateway-infra(039d2ff8-52e4-4560-8de5-cb8d0826718b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b"
Jul 17 20:47:28.880883 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.880863    2572 generic.go:356] "Generic (PLEG): container finished" podID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerID="eee38bc6653d5538d3cc7626ff4993c2c8b7fa5627957b375da857f1ce68c2bb" exitCode=6
Jul 17 20:47:28.880982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.880887    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" event={"ID":"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba","Type":"ContainerDied","Data":"eee38bc6653d5538d3cc7626ff4993c2c8b7fa5627957b375da857f1ce68c2bb"}
Jul 17 20:47:28.881136 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.881121    2572 scope.go:117] "RemoveContainer" containerID="eee38bc6653d5538d3cc7626ff4993c2c8b7fa5627957b375da857f1ce68c2bb"
Jul 17 20:47:28.881330 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:28.881295    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s_odh-ai-gateway-infra(d80d0b6a-4df8-49a1-8f6f-887707b3a9ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba"
Jul 17 20:47:28.889729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.889712    2572 scope.go:117] "RemoveContainer" containerID="d020029a9e88503edbad280b18d8ebabe03ea3dfcfd69e3bb7ac03b41c60517f"
Jul 17 20:47:28.900180 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:28.900138    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" podStartSLOduration=1.900126998 podStartE2EDuration="1.900126998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:47:28.899155814 +0000 UTC m=+3083.593131962" watchObservedRunningTime="2026-07-17 20:47:28.900126998 +0000 UTC m=+3083.594103144"
Jul 17 20:47:29.893374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:29.893341    2572 generic.go:356] "Generic (PLEG): container finished" podID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerID="b541c3715855ceb400fd009dd74db6cc1d564ea63ab1445a23be2234a72ab53f" exitCode=6
Jul 17 20:47:29.893809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:29.893432    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" event={"ID":"abff9fe3-c452-41f5-a010-846a0abf05ba","Type":"ContainerDied","Data":"b541c3715855ceb400fd009dd74db6cc1d564ea63ab1445a23be2234a72ab53f"}
Jul 17 20:47:29.893809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:29.893626    2572 scope.go:117] "RemoveContainer" containerID="b541c3715855ceb400fd009dd74db6cc1d564ea63ab1445a23be2234a72ab53f"
Jul 17 20:47:29.896843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:29.896821    2572 scope.go:117] "RemoveContainer" containerID="eee38bc6653d5538d3cc7626ff4993c2c8b7fa5627957b375da857f1ce68c2bb"
Jul 17 20:47:29.897083 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:29.897027    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s_odh-ai-gateway-infra(d80d0b6a-4df8-49a1-8f6f-887707b3a9ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba"
Jul 17 20:47:29.951360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:29.951323    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"]
Jul 17 20:47:30.088445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.088423    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"
Jul 17 20:47:30.127012 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.126983    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039d2ff8-52e4-4560-8de5-cb8d0826718b-maas-api-service-ca\") pod \"039d2ff8-52e4-4560-8de5-cb8d0826718b\" (UID: \"039d2ff8-52e4-4560-8de5-cb8d0826718b\") "
Jul 17 20:47:30.127410 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.127383    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039d2ff8-52e4-4560-8de5-cb8d0826718b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "039d2ff8-52e4-4560-8de5-cb8d0826718b" (UID: "039d2ff8-52e4-4560-8de5-cb8d0826718b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:47:30.228217 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.228188    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039d2ff8-52e4-4560-8de5-cb8d0826718b-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:47:30.902775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.902692    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"
Jul 17 20:47:30.902775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.902730    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t" event={"ID":"039d2ff8-52e4-4560-8de5-cb8d0826718b","Type":"ContainerDied","Data":"f6a3f69d4a2dc61a2da3631a26b0ef3353acbd9ccba2fdd01d2fefb554638504"}
Jul 17 20:47:30.903294 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.902797    2572 scope.go:117] "RemoveContainer" containerID="4a85f0ab714294390bb936118a54483a96f185b74aa32b7d1348e8d369041c70"
Jul 17 20:47:30.904818 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.904793    2572 generic.go:356] "Generic (PLEG): container finished" podID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerID="9bd50527db0b38cf7624ee67669ad4a869dbd97809766a5a0e5b4f13f34d041e" exitCode=6
Jul 17 20:47:30.904924 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.904843    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" event={"ID":"abff9fe3-c452-41f5-a010-846a0abf05ba","Type":"ContainerDied","Data":"9bd50527db0b38cf7624ee67669ad4a869dbd97809766a5a0e5b4f13f34d041e"}
Jul 17 20:47:30.905132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.905116    2572 scope.go:117] "RemoveContainer" containerID="9bd50527db0b38cf7624ee67669ad4a869dbd97809766a5a0e5b4f13f34d041e"
Jul 17 20:47:30.905357 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:30.905336    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl_odh-ai-gateway-infra(abff9fe3-c452-41f5-a010-846a0abf05ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba"
Jul 17 20:47:30.911587 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.911573    2572 scope.go:117] "RemoveContainer" containerID="b541c3715855ceb400fd009dd74db6cc1d564ea63ab1445a23be2234a72ab53f"
Jul 17 20:47:30.937561 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.937528    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"]
Jul 17 20:47:30.939556 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:30.939533    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-9x88t"]
Jul 17 20:47:31.854003 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:31.853967    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" path="/var/lib/kubelet/pods/039d2ff8-52e4-4560-8de5-cb8d0826718b/volumes"
Jul 17 20:47:31.912233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:31.912208    2572 scope.go:117] "RemoveContainer" containerID="9bd50527db0b38cf7624ee67669ad4a869dbd97809766a5a0e5b4f13f34d041e"
Jul 17 20:47:31.912600 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:31.912377    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl_odh-ai-gateway-infra(abff9fe3-c452-41f5-a010-846a0abf05ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba"
Jul 17 20:47:39.776203 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776167    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"]
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776499    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776510    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776520    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776525    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776535    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776542    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776600    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776606    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.776724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.776616    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="039d2ff8-52e4-4560-8de5-cb8d0826718b" containerName="revoke-keys"
Jul 17 20:47:39.780790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.780774    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"
Jul 17 20:47:39.786262 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.786235    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"]
Jul 17 20:47:39.906178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:39.906142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dadf8de-b997-4a64-b0c3-9744f31309b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s\" (UID: \"5dadf8de-b997-4a64-b0c3-9744f31309b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"
Jul 17 20:47:40.006770 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.006741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dadf8de-b997-4a64-b0c3-9744f31309b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s\" (UID: \"5dadf8de-b997-4a64-b0c3-9744f31309b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"
Jul 17 20:47:40.007416 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.007395    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dadf8de-b997-4a64-b0c3-9744f31309b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s\" (UID: \"5dadf8de-b997-4a64-b0c3-9744f31309b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"
Jul 17 20:47:40.092799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.092715    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"
Jul 17 20:47:40.223165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.223135    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"]
Jul 17 20:47:40.224721 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:47:40.224693    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dadf8de_b997_4a64_b0c3_9744f31309b2.slice/crio-1faafce7dc3fb7203b7269a02f5837258bab5cde2764dafb1c76f28a13c09975 WatchSource:0}: Error finding container 1faafce7dc3fb7203b7269a02f5837258bab5cde2764dafb1c76f28a13c09975: Status 404 returned error can't find the container with id 1faafce7dc3fb7203b7269a02f5837258bab5cde2764dafb1c76f28a13c09975
Jul 17 20:47:40.956293 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.956265    2572 generic.go:356] "Generic (PLEG): container finished" podID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerID="c713c126675abf58ea01ddb5f1f29e55dbc4d5c2deb708e9968d7a7b51dd54cd" exitCode=6
Jul 17 20:47:40.956613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.956354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" event={"ID":"5dadf8de-b997-4a64-b0c3-9744f31309b2","Type":"ContainerDied","Data":"c713c126675abf58ea01ddb5f1f29e55dbc4d5c2deb708e9968d7a7b51dd54cd"}
Jul 17 20:47:40.956613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.956438    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" event={"ID":"5dadf8de-b997-4a64-b0c3-9744f31309b2","Type":"ContainerStarted","Data":"1faafce7dc3fb7203b7269a02f5837258bab5cde2764dafb1c76f28a13c09975"}
Jul 17 20:47:40.956613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:40.956506    2572 scope.go:117] "RemoveContainer" containerID="c713c126675abf58ea01ddb5f1f29e55dbc4d5c2deb708e9968d7a7b51dd54cd"
Jul 17 20:47:41.848166 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:41.848087    2572 scope.go:117] "RemoveContainer" containerID="eee38bc6653d5538d3cc7626ff4993c2c8b7fa5627957b375da857f1ce68c2bb"
Jul 17 20:47:41.962952 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:41.962917    2572 generic.go:356] "Generic (PLEG): container finished" podID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerID="8a11a2eaebfe5fcb9ea94fcc3826290ad745e17016726b1296f993d0a410506e" exitCode=6
Jul 17 20:47:41.963387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:41.963004    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" event={"ID":"5dadf8de-b997-4a64-b0c3-9744f31309b2","Type":"ContainerDied","Data":"8a11a2eaebfe5fcb9ea94fcc3826290ad745e17016726b1296f993d0a410506e"}
Jul 17 20:47:41.963387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:41.963089    2572 scope.go:117] "RemoveContainer" containerID="c713c126675abf58ea01ddb5f1f29e55dbc4d5c2deb708e9968d7a7b51dd54cd"
Jul 17 20:47:41.963387 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:41.963212    2572 scope.go:117] "RemoveContainer" containerID="8a11a2eaebfe5fcb9ea94fcc3826290ad745e17016726b1296f993d0a410506e"
Jul 17 20:47:41.963551 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:41.963464    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s_odh-ai-gateway-infra(5dadf8de-b997-4a64-b0c3-9744f31309b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2"
Jul 17 20:47:42.791178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:42.791141    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:47:42.969349 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:42.969317    2572 generic.go:356] "Generic (PLEG): container finished" podID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerID="6fd362d853159128a8429e2738f5c953ab186a0983fc5e25110467f4df4a9b3f" exitCode=6
Jul 17 20:47:42.969814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:42.969387    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" event={"ID":"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba","Type":"ContainerDied","Data":"6fd362d853159128a8429e2738f5c953ab186a0983fc5e25110467f4df4a9b3f"}
Jul 17 20:47:42.969814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:42.969449    2572 scope.go:117] "RemoveContainer" containerID="eee38bc6653d5538d3cc7626ff4993c2c8b7fa5627957b375da857f1ce68c2bb"
Jul 17 20:47:42.969814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:42.969771    2572 scope.go:117] "RemoveContainer" containerID="6fd362d853159128a8429e2738f5c953ab186a0983fc5e25110467f4df4a9b3f"
Jul 17 20:47:42.970013 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:42.969990    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s_odh-ai-gateway-infra(d80d0b6a-4df8-49a1-8f6f-887707b3a9ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba"
Jul 17 20:47:42.971157 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:42.971143    2572 scope.go:117] "RemoveContainer" containerID="8a11a2eaebfe5fcb9ea94fcc3826290ad745e17016726b1296f993d0a410506e"
Jul 17 20:47:42.971312 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:42.971296    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s_odh-ai-gateway-infra(5dadf8de-b997-4a64-b0c3-9744f31309b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2"
Jul 17 20:47:43.847890 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:43.847858    2572 scope.go:117] "RemoveContainer" containerID="9bd50527db0b38cf7624ee67669ad4a869dbd97809766a5a0e5b4f13f34d041e"
Jul 17 20:47:43.994137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:43.994107    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"]
Jul 17 20:47:44.131081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.131058    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"
Jul 17 20:47:44.241937 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.241906    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80d0b6a-4df8-49a1-8f6f-887707b3a9ba-maas-api-service-ca\") pod \"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba\" (UID: \"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba\") "
Jul 17 20:47:44.242353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.242330    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80d0b6a-4df8-49a1-8f6f-887707b3a9ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" (UID: "d80d0b6a-4df8-49a1-8f6f-887707b3a9ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:47:44.343467 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.343431    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80d0b6a-4df8-49a1-8f6f-887707b3a9ba-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:47:44.984151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.984115    2572 generic.go:356] "Generic (PLEG): container finished" podID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerID="d37bcebe24b23a5fe8472d91ff001f2ee9e50bf193469f971d064290bd0c3536" exitCode=6
Jul 17 20:47:44.984347 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.984192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" event={"ID":"abff9fe3-c452-41f5-a010-846a0abf05ba","Type":"ContainerDied","Data":"d37bcebe24b23a5fe8472d91ff001f2ee9e50bf193469f971d064290bd0c3536"}
Jul 17 20:47:44.984347 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.984258    2572 scope.go:117] "RemoveContainer" containerID="9bd50527db0b38cf7624ee67669ad4a869dbd97809766a5a0e5b4f13f34d041e"
Jul 17 20:47:44.984613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.984596    2572 scope.go:117] "RemoveContainer" containerID="d37bcebe24b23a5fe8472d91ff001f2ee9e50bf193469f971d064290bd0c3536"
Jul 17 20:47:44.984808 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:44.984786    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl_odh-ai-gateway-infra(abff9fe3-c452-41f5-a010-846a0abf05ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba"
Jul 17 20:47:44.985773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.985743    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s" event={"ID":"d80d0b6a-4df8-49a1-8f6f-887707b3a9ba","Type":"ContainerDied","Data":"60a1317b11d5e814da6cfdff0d19926f583fa457e3fd45912b1b5b6021d2aad2"}
Jul 17 20:47:44.985872 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.985746    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"
Jul 17 20:47:44.995541 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:44.995380    2572 scope.go:117] "RemoveContainer" containerID="6fd362d853159128a8429e2738f5c953ab186a0983fc5e25110467f4df4a9b3f"
Jul 17 20:47:45.014623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:45.014593    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"]
Jul 17 20:47:45.016561 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:45.016540    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-dqb9s"]
Jul 17 20:47:45.854363 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:45.854330    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" path="/var/lib/kubelet/pods/d80d0b6a-4df8-49a1-8f6f-887707b3a9ba/volumes"
Jul 17 20:47:46.009305 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.009270    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"]
Jul 17 20:47:46.148995 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.148970    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"
Jul 17 20:47:46.259470 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.259441    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abff9fe3-c452-41f5-a010-846a0abf05ba-maas-api-service-ca\") pod \"abff9fe3-c452-41f5-a010-846a0abf05ba\" (UID: \"abff9fe3-c452-41f5-a010-846a0abf05ba\") "
Jul 17 20:47:46.259839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.259817    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abff9fe3-c452-41f5-a010-846a0abf05ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abff9fe3-c452-41f5-a010-846a0abf05ba" (UID: "abff9fe3-c452-41f5-a010-846a0abf05ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:47:46.360468 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.360436    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abff9fe3-c452-41f5-a010-846a0abf05ba-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:47:46.998608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.998572    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl" event={"ID":"abff9fe3-c452-41f5-a010-846a0abf05ba","Type":"ContainerDied","Data":"ffa236f826e52692ffc96ad9d77d65b3c77f73ffb938f6b7f59c1cc62d2e28da"}
Jul 17 20:47:46.998608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.998607    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"
Jul 17 20:47:46.998861 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:46.998630    2572 scope.go:117] "RemoveContainer" containerID="d37bcebe24b23a5fe8472d91ff001f2ee9e50bf193469f971d064290bd0c3536"
Jul 17 20:47:47.024210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:47.024181    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"]
Jul 17 20:47:47.032794 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:47.032768    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-q5lsl"]
Jul 17 20:47:47.853510 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:47.853474    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" path="/var/lib/kubelet/pods/abff9fe3-c452-41f5-a010-846a0abf05ba/volumes"
Jul 17 20:47:57.848477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:57.848441    2572 scope.go:117] "RemoveContainer" containerID="8a11a2eaebfe5fcb9ea94fcc3826290ad745e17016726b1296f993d0a410506e"
Jul 17 20:47:59.067527 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:59.067493    2572 generic.go:356] "Generic (PLEG): container finished" podID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerID="7c36ee2877d643c3dd8baa489d0c6ababe164986e5c41c72d68712dc11e6bf74" exitCode=6
Jul 17 20:47:59.067973 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:59.067553    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" event={"ID":"5dadf8de-b997-4a64-b0c3-9744f31309b2","Type":"ContainerDied","Data":"7c36ee2877d643c3dd8baa489d0c6ababe164986e5c41c72d68712dc11e6bf74"}
Jul 17 20:47:59.067973 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:59.067614    2572 scope.go:117] "RemoveContainer" containerID="8a11a2eaebfe5fcb9ea94fcc3826290ad745e17016726b1296f993d0a410506e"
Jul 17 20:47:59.067973 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:59.067898    2572 scope.go:117] "RemoveContainer" containerID="7c36ee2877d643c3dd8baa489d0c6ababe164986e5c41c72d68712dc11e6bf74"
Jul 17 20:47:59.068137 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:47:59.068097    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s_odh-ai-gateway-infra(5dadf8de-b997-4a64-b0c3-9744f31309b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2"
Jul 17 20:47:59.497329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:47:59.497287    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:48:00.096143 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:00.096112    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"]
Jul 17 20:48:00.225149 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:00.225123    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"
Jul 17 20:48:00.377722 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:00.377639    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dadf8de-b997-4a64-b0c3-9744f31309b2-maas-api-service-ca\") pod \"5dadf8de-b997-4a64-b0c3-9744f31309b2\" (UID: \"5dadf8de-b997-4a64-b0c3-9744f31309b2\") "
Jul 17 20:48:00.377991 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:00.377969    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dadf8de-b997-4a64-b0c3-9744f31309b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5dadf8de-b997-4a64-b0c3-9744f31309b2" (UID: "5dadf8de-b997-4a64-b0c3-9744f31309b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:48:00.478875 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:00.478845    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dadf8de-b997-4a64-b0c3-9744f31309b2-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:48:01.079700 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.079669    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s" event={"ID":"5dadf8de-b997-4a64-b0c3-9744f31309b2","Type":"ContainerDied","Data":"1faafce7dc3fb7203b7269a02f5837258bab5cde2764dafb1c76f28a13c09975"}
Jul 17 20:48:01.079700 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.079695    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"
Jul 17 20:48:01.079930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.079728    2572 scope.go:117] "RemoveContainer" containerID="7c36ee2877d643c3dd8baa489d0c6ababe164986e5c41c72d68712dc11e6bf74"
Jul 17 20:48:01.100498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.100470    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"]
Jul 17 20:48:01.106171 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.106149    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-4dz9s"]
Jul 17 20:48:01.684236 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684198    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"]
Jul 17 20:48:01.684540 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684528    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:01.684585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684542    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:01.684585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684551    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:01.684585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684556    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:01.684585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684564    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:01.684585 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684571    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684586    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684594    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684617    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684625    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684635    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684640    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684687    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684694    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684701    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684708    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684713    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684719    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684724    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:01.684754 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.684729    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:01.688815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.688798    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"
Jul 17 20:48:01.691235 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.691218    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:48:01.695368 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.695349    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"]
Jul 17 20:48:01.789097 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.789062    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef11352b-6ef0-4642-82f2-1c5ae7f56e8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8qqfd\" (UID: \"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"
Jul 17 20:48:01.853021 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.852979    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" path="/var/lib/kubelet/pods/5dadf8de-b997-4a64-b0c3-9744f31309b2/volumes"
Jul 17 20:48:01.889550 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.889525    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef11352b-6ef0-4642-82f2-1c5ae7f56e8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8qqfd\" (UID: \"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"
Jul 17 20:48:01.890126 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:01.890108    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef11352b-6ef0-4642-82f2-1c5ae7f56e8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-8qqfd\" (UID: \"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"
Jul 17 20:48:02.000356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:02.000328    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"
Jul 17 20:48:02.129951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:02.129925    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"]
Jul 17 20:48:02.131998 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:48:02.131967    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef11352b_6ef0_4642_82f2_1c5ae7f56e8d.slice/crio-dbb8f50314ca16043df398fd19511bcfa0ee56fee5ab1400fbb3ba52c34d3ab0 WatchSource:0}: Error finding container dbb8f50314ca16043df398fd19511bcfa0ee56fee5ab1400fbb3ba52c34d3ab0: Status 404 returned error can't find the container with id dbb8f50314ca16043df398fd19511bcfa0ee56fee5ab1400fbb3ba52c34d3ab0
Jul 17 20:48:03.093555 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:03.093521    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerID="4f4b87d4b59e6f4e87bb0823534818f5ada3aa9c3e526debfe1cf106cc46ac5c" exitCode=6
Jul 17 20:48:03.093736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:03.093607    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" event={"ID":"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d","Type":"ContainerDied","Data":"4f4b87d4b59e6f4e87bb0823534818f5ada3aa9c3e526debfe1cf106cc46ac5c"}
Jul 17 20:48:03.093736 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:03.093684    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" event={"ID":"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d","Type":"ContainerStarted","Data":"dbb8f50314ca16043df398fd19511bcfa0ee56fee5ab1400fbb3ba52c34d3ab0"}
Jul 17 20:48:03.093820 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:03.093761    2572 scope.go:117] "RemoveContainer" containerID="4f4b87d4b59e6f4e87bb0823534818f5ada3aa9c3e526debfe1cf106cc46ac5c"
Jul 17 20:48:03.641529 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:03.641499    2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef11352b_6ef0_4642_82f2_1c5ae7f56e8d.slice/crio-conmon-2ce6a162bfa531b8e1a008804ae051216fcf4f98e9d734a438084a433c0b6fec.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 20:48:04.100313 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:04.100282    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerID="2ce6a162bfa531b8e1a008804ae051216fcf4f98e9d734a438084a433c0b6fec" exitCode=6
Jul 17 20:48:04.100468 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:04.100351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" event={"ID":"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d","Type":"ContainerDied","Data":"2ce6a162bfa531b8e1a008804ae051216fcf4f98e9d734a438084a433c0b6fec"}
Jul 17 20:48:04.100468 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:04.100403    2572 scope.go:117] "RemoveContainer" containerID="4f4b87d4b59e6f4e87bb0823534818f5ada3aa9c3e526debfe1cf106cc46ac5c"
Jul 17 20:48:04.100599 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:04.100585    2572 scope.go:117] "RemoveContainer" containerID="2ce6a162bfa531b8e1a008804ae051216fcf4f98e9d734a438084a433c0b6fec"
Jul 17 20:48:04.100838 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:04.100809    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8qqfd_odh-ai-gateway-infra(ef11352b-6ef0-4642-82f2-1c5ae7f56e8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d"
Jul 17 20:48:05.106737 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:05.106698    2572 scope.go:117] "RemoveContainer" containerID="2ce6a162bfa531b8e1a008804ae051216fcf4f98e9d734a438084a433c0b6fec"
Jul 17 20:48:05.107235 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:05.106924    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8qqfd_odh-ai-gateway-infra(ef11352b-6ef0-4642-82f2-1c5ae7f56e8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d"
Jul 17 20:48:15.306496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.306467    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:48:15.787155 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787120    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"]
Jul 17 20:48:15.787480 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787468    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:15.787526 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787481    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dadf8de-b997-4a64-b0c3-9744f31309b2" containerName="revoke-keys"
Jul 17 20:48:15.787526 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787491    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:15.787526 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787496    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="abff9fe3-c452-41f5-a010-846a0abf05ba" containerName="revoke-keys"
Jul 17 20:48:15.787526 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787506    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:15.787526 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787512    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:15.787729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.787556    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d80d0b6a-4df8-49a1-8f6f-887707b3a9ba" containerName="revoke-keys"
Jul 17 20:48:15.792150 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.792126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"
Jul 17 20:48:15.804070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.804022    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"]
Jul 17 20:48:15.850526 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.850501    2572 scope.go:117] "RemoveContainer" containerID="2ce6a162bfa531b8e1a008804ae051216fcf4f98e9d734a438084a433c0b6fec"
Jul 17 20:48:15.991885 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:15.991845    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fc080-0d95-4a1e-ae31-7d4488c5e174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l\" (UID: \"d87fc080-0d95-4a1e-ae31-7d4488c5e174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"
Jul 17 20:48:16.093092 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:16.093029    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fc080-0d95-4a1e-ae31-7d4488c5e174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l\" (UID: \"d87fc080-0d95-4a1e-ae31-7d4488c5e174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"
Jul 17 20:48:16.093711 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:16.093691    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fc080-0d95-4a1e-ae31-7d4488c5e174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l\" (UID: \"d87fc080-0d95-4a1e-ae31-7d4488c5e174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"
Jul 17 20:48:16.101942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:16.101919    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"
Jul 17 20:48:16.164311 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:16.164167    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" event={"ID":"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d","Type":"ContainerStarted","Data":"25652b6f45d83710638e3607dde0d75133a6ab0ff85e3aeb713fc654c662934e"}
Jul 17 20:48:16.182362 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:16.182312    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" podStartSLOduration=15.182295147 podStartE2EDuration="15.182295147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:48:16.181396282 +0000 UTC m=+3130.875372430" watchObservedRunningTime="2026-07-17 20:48:16.182295147 +0000 UTC m=+3130.876271325"
Jul 17 20:48:16.236896 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:16.236872    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"]
Jul 17 20:48:16.239313 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:48:16.239284    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87fc080_0d95_4a1e_ae31_7d4488c5e174.slice/crio-06cf503ca819076e9026dc8c9053fcdb8f83ca33796c0a5c4a81755fc9925199 WatchSource:0}: Error finding container 06cf503ca819076e9026dc8c9053fcdb8f83ca33796c0a5c4a81755fc9925199: Status 404 returned error can't find the container with id 06cf503ca819076e9026dc8c9053fcdb8f83ca33796c0a5c4a81755fc9925199
Jul 17 20:48:17.170771 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.170689    2572 generic.go:356] "Generic (PLEG): container finished" podID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerID="7fd77e0a20da122940d9c94cb72bd64a6208cf3b038fef9a004563a823f481c8" exitCode=6
Jul 17 20:48:17.171213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.170780    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" event={"ID":"d87fc080-0d95-4a1e-ae31-7d4488c5e174","Type":"ContainerDied","Data":"7fd77e0a20da122940d9c94cb72bd64a6208cf3b038fef9a004563a823f481c8"}
Jul 17 20:48:17.171213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.170839    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" event={"ID":"d87fc080-0d95-4a1e-ae31-7d4488c5e174","Type":"ContainerStarted","Data":"06cf503ca819076e9026dc8c9053fcdb8f83ca33796c0a5c4a81755fc9925199"}
Jul 17 20:48:17.171213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.170962    2572 scope.go:117] "RemoveContainer" containerID="7fd77e0a20da122940d9c94cb72bd64a6208cf3b038fef9a004563a823f481c8"
Jul 17 20:48:17.172769 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.172749    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerID="25652b6f45d83710638e3607dde0d75133a6ab0ff85e3aeb713fc654c662934e" exitCode=6
Jul 17 20:48:17.172851 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.172779    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" event={"ID":"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d","Type":"ContainerDied","Data":"25652b6f45d83710638e3607dde0d75133a6ab0ff85e3aeb713fc654c662934e"}
Jul 17 20:48:17.172851 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.172812    2572 scope.go:117] "RemoveContainer" containerID="2ce6a162bfa531b8e1a008804ae051216fcf4f98e9d734a438084a433c0b6fec"
Jul 17 20:48:17.173026 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.173014    2572 scope.go:117] "RemoveContainer" containerID="25652b6f45d83710638e3607dde0d75133a6ab0ff85e3aeb713fc654c662934e"
Jul 17 20:48:17.173226 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:17.173204    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-8qqfd_odh-ai-gateway-infra(ef11352b-6ef0-4642-82f2-1c5ae7f56e8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d"
Jul 17 20:48:17.199873 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.199845    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"]
Jul 17 20:48:17.984472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.984432    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"]
Jul 17 20:48:17.987741 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.987722    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"
Jul 17 20:48:17.995251 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:17.995161    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"]
Jul 17 20:48:18.004569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.004541    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-nnckt\" (UID: \"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"
Jul 17 20:48:18.105296 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.105215    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-nnckt\" (UID: \"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"
Jul 17 20:48:18.105882 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.105863    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-nnckt\" (UID: \"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"
Jul 17 20:48:18.179906 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.179874    2572 generic.go:356] "Generic (PLEG): container finished" podID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerID="237fd939dd0ded44122e344ddb773651caa8eb0cd2f4da590afb7dad8a6a5512" exitCode=6
Jul 17 20:48:18.180402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.179963    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" event={"ID":"d87fc080-0d95-4a1e-ae31-7d4488c5e174","Type":"ContainerDied","Data":"237fd939dd0ded44122e344ddb773651caa8eb0cd2f4da590afb7dad8a6a5512"}
Jul 17 20:48:18.180402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.180027    2572 scope.go:117] "RemoveContainer" containerID="7fd77e0a20da122940d9c94cb72bd64a6208cf3b038fef9a004563a823f481c8"
Jul 17 20:48:18.180402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.180280    2572 scope.go:117] "RemoveContainer" containerID="237fd939dd0ded44122e344ddb773651caa8eb0cd2f4da590afb7dad8a6a5512"
Jul 17 20:48:18.180554 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:18.180484    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l_odh-ai-gateway-infra(d87fc080-0d95-4a1e-ae31-7d4488c5e174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174"
Jul 17 20:48:18.299462 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.299441    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"
Jul 17 20:48:18.306404 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.306385    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"
Jul 17 20:48:18.407391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.407362    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef11352b-6ef0-4642-82f2-1c5ae7f56e8d-maas-api-service-ca\") pod \"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d\" (UID: \"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d\") "
Jul 17 20:48:18.407710 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.407688    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef11352b-6ef0-4642-82f2-1c5ae7f56e8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" (UID: "ef11352b-6ef0-4642-82f2-1c5ae7f56e8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:48:18.457410 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.457386    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"]
Jul 17 20:48:18.459664 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:48:18.459641    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e96e2a1_7cda_4615_bcb3_3eb42e3c4a68.slice/crio-b222da1b5274334cf3fdbcde1bd0f29c5410c05e06e6e162b05c99a14482fa98 WatchSource:0}: Error finding container b222da1b5274334cf3fdbcde1bd0f29c5410c05e06e6e162b05c99a14482fa98: Status 404 returned error can't find the container with id b222da1b5274334cf3fdbcde1bd0f29c5410c05e06e6e162b05c99a14482fa98
Jul 17 20:48:18.508085 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:18.508012    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef11352b-6ef0-4642-82f2-1c5ae7f56e8d-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:48:19.188835 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.188795    2572 scope.go:117] "RemoveContainer" containerID="237fd939dd0ded44122e344ddb773651caa8eb0cd2f4da590afb7dad8a6a5512"
Jul 17 20:48:19.189302 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:19.189025    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l_odh-ai-gateway-infra(d87fc080-0d95-4a1e-ae31-7d4488c5e174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174"
Jul 17 20:48:19.189988 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.189967    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"
Jul 17 20:48:19.190090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.189985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd" event={"ID":"ef11352b-6ef0-4642-82f2-1c5ae7f56e8d","Type":"ContainerDied","Data":"dbb8f50314ca16043df398fd19511bcfa0ee56fee5ab1400fbb3ba52c34d3ab0"}
Jul 17 20:48:19.190199 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.190104    2572 scope.go:117] "RemoveContainer" containerID="25652b6f45d83710638e3607dde0d75133a6ab0ff85e3aeb713fc654c662934e"
Jul 17 20:48:19.191623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.191593    2572 generic.go:356] "Generic (PLEG): container finished" podID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerID="ba60dc44340052079dc8c92d5652ae83646a19fa384d845585e683eb762da376" exitCode=6
Jul 17 20:48:19.191745 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.191639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" event={"ID":"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68","Type":"ContainerDied","Data":"ba60dc44340052079dc8c92d5652ae83646a19fa384d845585e683eb762da376"}
Jul 17 20:48:19.191745 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.191692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" event={"ID":"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68","Type":"ContainerStarted","Data":"b222da1b5274334cf3fdbcde1bd0f29c5410c05e06e6e162b05c99a14482fa98"}
Jul 17 20:48:19.191829 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.191817    2572 scope.go:117] "RemoveContainer" containerID="ba60dc44340052079dc8c92d5652ae83646a19fa384d845585e683eb762da376"
Jul 17 20:48:19.238218 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.238185    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"]
Jul 17 20:48:19.248209 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.248173    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-8qqfd"]
Jul 17 20:48:19.853010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:19.852979    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" path="/var/lib/kubelet/pods/ef11352b-6ef0-4642-82f2-1c5ae7f56e8d/volumes"
Jul 17 20:48:20.208664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:20.208568    2572 generic.go:356] "Generic (PLEG): container finished" podID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerID="1275e5b5f62e0ca061d153530b42ca9b2bc5d7f9f34bd14f8405bc72e4a98ede" exitCode=6
Jul 17 20:48:20.208664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:20.208630    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" event={"ID":"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68","Type":"ContainerDied","Data":"1275e5b5f62e0ca061d153530b42ca9b2bc5d7f9f34bd14f8405bc72e4a98ede"}
Jul 17 20:48:20.209207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:20.208697    2572 scope.go:117] "RemoveContainer" containerID="ba60dc44340052079dc8c92d5652ae83646a19fa384d845585e683eb762da376"
Jul 17 20:48:20.209207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:20.208867    2572 scope.go:117] "RemoveContainer" containerID="1275e5b5f62e0ca061d153530b42ca9b2bc5d7f9f34bd14f8405bc72e4a98ede"
Jul 17 20:48:20.209207 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:20.209109    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-nnckt_odh-ai-gateway-infra(3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68"
Jul 17 20:48:21.215183 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:21.215154    2572 scope.go:117] "RemoveContainer" containerID="1275e5b5f62e0ca061d153530b42ca9b2bc5d7f9f34bd14f8405bc72e4a98ede"
Jul 17 20:48:21.215568 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:21.215322    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-nnckt_odh-ai-gateway-infra(3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68"
Jul 17 20:48:29.603457 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.603367    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kt8sb"]
Jul 17 20:48:29.603914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.603845    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.603914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.603864    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.603914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.603886    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.603914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.603895    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.603914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.603908    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.604226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.603919    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.604226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.604000    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.604226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.604013    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.604226 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.604022    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef11352b-6ef0-4642-82f2-1c5ae7f56e8d" containerName="revoke-keys"
Jul 17 20:48:29.608275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.608251    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:29.611305 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.611285    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bz865\""
Jul 17 20:48:29.618445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.618424    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kt8sb"]
Jul 17 20:48:29.690225 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.690196    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcb2\" (UniqueName: \"kubernetes.io/projected/76fc410f-af3c-4e6c-80c9-4f526d991a93-kube-api-access-zmcb2\") pod \"kuadrant-operator-catalog-kt8sb\" (UID: \"76fc410f-af3c-4e6c-80c9-4f526d991a93\") " pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:29.791438 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.791402    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcb2\" (UniqueName: \"kubernetes.io/projected/76fc410f-af3c-4e6c-80c9-4f526d991a93-kube-api-access-zmcb2\") pod \"kuadrant-operator-catalog-kt8sb\" (UID: \"76fc410f-af3c-4e6c-80c9-4f526d991a93\") " pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:29.800932 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.800892    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcb2\" (UniqueName: \"kubernetes.io/projected/76fc410f-af3c-4e6c-80c9-4f526d991a93-kube-api-access-zmcb2\") pod \"kuadrant-operator-catalog-kt8sb\" (UID: \"76fc410f-af3c-4e6c-80c9-4f526d991a93\") " pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:29.917151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:29.917068    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:30.046250 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:30.046222    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kt8sb"]
Jul 17 20:48:30.048362 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:48:30.048337    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fc410f_af3c_4e6c_80c9_4f526d991a93.slice/crio-e77fc302bc61fe1696e69e8bcd3df377057f54748d2a723f19d8761c1d6b4325 WatchSource:0}: Error finding container e77fc302bc61fe1696e69e8bcd3df377057f54748d2a723f19d8761c1d6b4325: Status 404 returned error can't find the container with id e77fc302bc61fe1696e69e8bcd3df377057f54748d2a723f19d8761c1d6b4325
Jul 17 20:48:30.049983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:30.049969    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 20:48:30.257990 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:30.257956    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb" event={"ID":"76fc410f-af3c-4e6c-80c9-4f526d991a93","Type":"ContainerStarted","Data":"e77fc302bc61fe1696e69e8bcd3df377057f54748d2a723f19d8761c1d6b4325"}
Jul 17 20:48:30.848977 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:30.848943    2572 scope.go:117] "RemoveContainer" containerID="237fd939dd0ded44122e344ddb773651caa8eb0cd2f4da590afb7dad8a6a5512"
Jul 17 20:48:31.268500 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:31.268463    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" event={"ID":"d87fc080-0d95-4a1e-ae31-7d4488c5e174","Type":"ContainerStarted","Data":"b0ebb97fbc0fff1494aa7e2e80861f9daa7adca183e2fb04bfd99744cc2fc335"}
Jul 17 20:48:31.286790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:31.286736    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" podStartSLOduration=16.286718055 podStartE2EDuration="16.286718055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:48:31.285399223 +0000 UTC m=+3145.979375370" watchObservedRunningTime="2026-07-17 20:48:31.286718055 +0000 UTC m=+3145.980694200"
Jul 17 20:48:32.274386 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.274355    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb" event={"ID":"76fc410f-af3c-4e6c-80c9-4f526d991a93","Type":"ContainerStarted","Data":"cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34"}
Jul 17 20:48:32.276151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.276129    2572 generic.go:356] "Generic (PLEG): container finished" podID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerID="b0ebb97fbc0fff1494aa7e2e80861f9daa7adca183e2fb04bfd99744cc2fc335" exitCode=6
Jul 17 20:48:32.276270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.276200    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" event={"ID":"d87fc080-0d95-4a1e-ae31-7d4488c5e174","Type":"ContainerDied","Data":"b0ebb97fbc0fff1494aa7e2e80861f9daa7adca183e2fb04bfd99744cc2fc335"}
Jul 17 20:48:32.276270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.276249    2572 scope.go:117] "RemoveContainer" containerID="237fd939dd0ded44122e344ddb773651caa8eb0cd2f4da590afb7dad8a6a5512"
Jul 17 20:48:32.276455 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.276442    2572 scope.go:117] "RemoveContainer" containerID="b0ebb97fbc0fff1494aa7e2e80861f9daa7adca183e2fb04bfd99744cc2fc335"
Jul 17 20:48:32.276657 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:32.276636    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l_odh-ai-gateway-infra(d87fc080-0d95-4a1e-ae31-7d4488c5e174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174"
Jul 17 20:48:32.291620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.291578    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb" podStartSLOduration=1.779388395 podStartE2EDuration="3.291565687s" totalImagesPullingTime="1.512177292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:48:32.289935015 +0000 UTC m=+3146.983911172" watchObservedRunningTime="2026-07-17 20:48:32.291565687 +0000 UTC m=+3146.985541832"
Jul 17 20:48:32.296041 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.296021    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"]
Jul 17 20:48:32.782336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.782296    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:48:32.848281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:32.848249    2572 scope.go:117] "RemoveContainer" containerID="1275e5b5f62e0ca061d153530b42ca9b2bc5d7f9f34bd14f8405bc72e4a98ede"
Jul 17 20:48:33.282269 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:33.282231    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" event={"ID":"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68","Type":"ContainerStarted","Data":"30b6e4aa964c8780cc8e432a5515eabc6c7e83fc29c383752e06f6df8cd077f9"}
Jul 17 20:48:33.302913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:33.302808    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" podStartSLOduration=16.302793229 podStartE2EDuration="16.302793229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:48:33.30186016 +0000 UTC m=+3147.995836307" watchObservedRunningTime="2026-07-17 20:48:33.302793229 +0000 UTC m=+3147.996769374"
Jul 17 20:48:33.422265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:33.422239    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"
Jul 17 20:48:33.520945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:33.520909    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fc080-0d95-4a1e-ae31-7d4488c5e174-maas-api-service-ca\") pod \"d87fc080-0d95-4a1e-ae31-7d4488c5e174\" (UID: \"d87fc080-0d95-4a1e-ae31-7d4488c5e174\") "
Jul 17 20:48:33.521286 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:33.521265    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87fc080-0d95-4a1e-ae31-7d4488c5e174-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d87fc080-0d95-4a1e-ae31-7d4488c5e174" (UID: "d87fc080-0d95-4a1e-ae31-7d4488c5e174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:48:33.622090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:33.621986    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fc080-0d95-4a1e-ae31-7d4488c5e174-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:48:34.289950 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.289920    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"
Jul 17 20:48:34.290428 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.289961    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l" event={"ID":"d87fc080-0d95-4a1e-ae31-7d4488c5e174","Type":"ContainerDied","Data":"06cf503ca819076e9026dc8c9053fcdb8f83ca33796c0a5c4a81755fc9925199"}
Jul 17 20:48:34.290428 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.290027    2572 scope.go:117] "RemoveContainer" containerID="b0ebb97fbc0fff1494aa7e2e80861f9daa7adca183e2fb04bfd99744cc2fc335"
Jul 17 20:48:34.291993 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.291965    2572 generic.go:356] "Generic (PLEG): container finished" podID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerID="30b6e4aa964c8780cc8e432a5515eabc6c7e83fc29c383752e06f6df8cd077f9" exitCode=6
Jul 17 20:48:34.292127 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.292041    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" event={"ID":"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68","Type":"ContainerDied","Data":"30b6e4aa964c8780cc8e432a5515eabc6c7e83fc29c383752e06f6df8cd077f9"}
Jul 17 20:48:34.292412 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.292393    2572 scope.go:117] "RemoveContainer" containerID="30b6e4aa964c8780cc8e432a5515eabc6c7e83fc29c383752e06f6df8cd077f9"
Jul 17 20:48:34.292709 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:34.292688    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-nnckt_odh-ai-gateway-infra(3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68"
Jul 17 20:48:34.298747 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.298731    2572 scope.go:117] "RemoveContainer" containerID="1275e5b5f62e0ca061d153530b42ca9b2bc5d7f9f34bd14f8405bc72e4a98ede"
Jul 17 20:48:34.305782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.305759    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"]
Jul 17 20:48:34.310174 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.310154    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-r8g7l"]
Jul 17 20:48:34.312905 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:34.312886    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"]
Jul 17 20:48:35.428900 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:35.428879    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"
Jul 17 20:48:35.536558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:35.536530    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68-maas-api-service-ca\") pod \"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68\" (UID: \"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68\") "
Jul 17 20:48:35.536869 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:35.536848    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" (UID: "3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:48:35.637950 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:35.637868    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:48:35.855181 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:35.855147    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" path="/var/lib/kubelet/pods/d87fc080-0d95-4a1e-ae31-7d4488c5e174/volumes"
Jul 17 20:48:36.308389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:36.308333    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt" event={"ID":"3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68","Type":"ContainerDied","Data":"b222da1b5274334cf3fdbcde1bd0f29c5410c05e06e6e162b05c99a14482fa98"}
Jul 17 20:48:36.308389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:36.308367    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"
Jul 17 20:48:36.308655 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:36.308512    2572 scope.go:117] "RemoveContainer" containerID="30b6e4aa964c8780cc8e432a5515eabc6c7e83fc29c383752e06f6df8cd077f9"
Jul 17 20:48:36.326977 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:36.326949    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"]
Jul 17 20:48:36.331024 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:36.330999    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-nnckt"]
Jul 17 20:48:37.853151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:37.853109    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" path="/var/lib/kubelet/pods/3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68/volumes"
Jul 17 20:48:39.826423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826387    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"]
Jul 17 20:48:39.826919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826859    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.826919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826874    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.826919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826887    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.826919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826895    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.826919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826904    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.826919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826912    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826925    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826933    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826952    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826960    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826974    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.826982    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.827071    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.827081    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.827093    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:39.827233 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.827103    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:39.831722 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.831701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"
Jul 17 20:48:39.834379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.834349    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:48:39.837185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.837154    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"]
Jul 17 20:48:39.868936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.868905    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f5c0b6-42a5-4107-83c2-dd78f57c2098-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn\" (UID: \"10f5c0b6-42a5-4107-83c2-dd78f57c2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"
Jul 17 20:48:39.917604 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.917556    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:39.917761 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.917639    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:39.940202 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.940174    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:39.969554 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.969520    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f5c0b6-42a5-4107-83c2-dd78f57c2098-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn\" (UID: \"10f5c0b6-42a5-4107-83c2-dd78f57c2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"
Jul 17 20:48:39.970219 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:39.970200    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f5c0b6-42a5-4107-83c2-dd78f57c2098-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn\" (UID: \"10f5c0b6-42a5-4107-83c2-dd78f57c2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"
Jul 17 20:48:40.143661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:40.143582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"
Jul 17 20:48:40.270802 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:40.270780    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"]
Jul 17 20:48:40.272590 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:48:40.272559    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f5c0b6_42a5_4107_83c2_dd78f57c2098.slice/crio-c8fcbf32a3c251399f945bc1111b665c4d643db6602e25b9632d02a7374ccf4a WatchSource:0}: Error finding container c8fcbf32a3c251399f945bc1111b665c4d643db6602e25b9632d02a7374ccf4a: Status 404 returned error can't find the container with id c8fcbf32a3c251399f945bc1111b665c4d643db6602e25b9632d02a7374ccf4a
Jul 17 20:48:40.330396 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:40.330359    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" event={"ID":"10f5c0b6-42a5-4107-83c2-dd78f57c2098","Type":"ContainerStarted","Data":"c8fcbf32a3c251399f945bc1111b665c4d643db6602e25b9632d02a7374ccf4a"}
Jul 17 20:48:40.359818 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:40.359793    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:40.395913 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:40.395829    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kt8sb"]
Jul 17 20:48:41.336748 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:41.336712    2572 generic.go:356] "Generic (PLEG): container finished" podID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerID="f27a8cf7e909433948402e73624e03828477b838e2c4412945445a00b61c1503" exitCode=6
Jul 17 20:48:41.337178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:41.336803    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" event={"ID":"10f5c0b6-42a5-4107-83c2-dd78f57c2098","Type":"ContainerDied","Data":"f27a8cf7e909433948402e73624e03828477b838e2c4412945445a00b61c1503"}
Jul 17 20:48:41.337178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:41.336951    2572 scope.go:117] "RemoveContainer" containerID="f27a8cf7e909433948402e73624e03828477b838e2c4412945445a00b61c1503"
Jul 17 20:48:42.343846 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.343800    2572 generic.go:356] "Generic (PLEG): container finished" podID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerID="fb2fef0ff74b57b9255c4c18bc3b02d283dd833b53776ac2937b244abfb2d60d" exitCode=6
Jul 17 20:48:42.344346 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.343892    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" event={"ID":"10f5c0b6-42a5-4107-83c2-dd78f57c2098","Type":"ContainerDied","Data":"fb2fef0ff74b57b9255c4c18bc3b02d283dd833b53776ac2937b244abfb2d60d"}
Jul 17 20:48:42.344346 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.343970    2572 scope.go:117] "RemoveContainer" containerID="f27a8cf7e909433948402e73624e03828477b838e2c4412945445a00b61c1503"
Jul 17 20:48:42.344346 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.343992    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb" podUID="76fc410f-af3c-4e6c-80c9-4f526d991a93" containerName="registry-server" containerID="cri-o://cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34" gracePeriod=2
Jul 17 20:48:42.344346 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.344245    2572 scope.go:117] "RemoveContainer" containerID="fb2fef0ff74b57b9255c4c18bc3b02d283dd833b53776ac2937b244abfb2d60d"
Jul 17 20:48:42.344810 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:42.344458    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn_odh-ai-gateway-infra(10f5c0b6-42a5-4107-83c2-dd78f57c2098)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098"
Jul 17 20:48:42.597007 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.596950    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:42.695655 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.695627    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmcb2\" (UniqueName: \"kubernetes.io/projected/76fc410f-af3c-4e6c-80c9-4f526d991a93-kube-api-access-zmcb2\") pod \"76fc410f-af3c-4e6c-80c9-4f526d991a93\" (UID: \"76fc410f-af3c-4e6c-80c9-4f526d991a93\") "
Jul 17 20:48:42.697672 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.697636    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fc410f-af3c-4e6c-80c9-4f526d991a93-kube-api-access-zmcb2" (OuterVolumeSpecName: "kube-api-access-zmcb2") pod "76fc410f-af3c-4e6c-80c9-4f526d991a93" (UID: "76fc410f-af3c-4e6c-80c9-4f526d991a93"). InnerVolumeSpecName "kube-api-access-zmcb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:48:42.796453 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:42.796417    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcb2\" (UniqueName: \"kubernetes.io/projected/76fc410f-af3c-4e6c-80c9-4f526d991a93-kube-api-access-zmcb2\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:48:43.350704 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.350676    2572 scope.go:117] "RemoveContainer" containerID="fb2fef0ff74b57b9255c4c18bc3b02d283dd833b53776ac2937b244abfb2d60d"
Jul 17 20:48:43.351142 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:43.350865    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn_odh-ai-gateway-infra(10f5c0b6-42a5-4107-83c2-dd78f57c2098)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098"
Jul 17 20:48:43.351782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.351759    2572 generic.go:356] "Generic (PLEG): container finished" podID="76fc410f-af3c-4e6c-80c9-4f526d991a93" containerID="cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34" exitCode=0
Jul 17 20:48:43.351902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.351796    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb"
Jul 17 20:48:43.351902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.351801    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb" event={"ID":"76fc410f-af3c-4e6c-80c9-4f526d991a93","Type":"ContainerDied","Data":"cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34"}
Jul 17 20:48:43.351902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.351831    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kt8sb" event={"ID":"76fc410f-af3c-4e6c-80c9-4f526d991a93","Type":"ContainerDied","Data":"e77fc302bc61fe1696e69e8bcd3df377057f54748d2a723f19d8761c1d6b4325"}
Jul 17 20:48:43.351902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.351848    2572 scope.go:117] "RemoveContainer" containerID="cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34"
Jul 17 20:48:43.360903 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.360885    2572 scope.go:117] "RemoveContainer" containerID="cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34"
Jul 17 20:48:43.361181 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:43.361162    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34\": container with ID starting with cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34 not found: ID does not exist" containerID="cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34"
Jul 17 20:48:43.361230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.361188    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34"} err="failed to get container status \"cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34\": rpc error: code = NotFound desc = could not find container \"cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34\": container with ID starting with cc05283fc31736824a0fa4b8bcc8af9642f610582383491da9197a98b4cbcb34 not found: ID does not exist"
Jul 17 20:48:43.383565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.383519    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kt8sb"]
Jul 17 20:48:43.385426 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.385405    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kt8sb"]
Jul 17 20:48:43.853553 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:43.853520    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fc410f-af3c-4e6c-80c9-4f526d991a93" path="/var/lib/kubelet/pods/76fc410f-af3c-4e6c-80c9-4f526d991a93/volumes"
Jul 17 20:48:48.083220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:48.083182    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:48:51.737865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.737833    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"]
Jul 17 20:48:51.738257 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.738209    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fc410f-af3c-4e6c-80c9-4f526d991a93" containerName="registry-server"
Jul 17 20:48:51.738257 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.738221    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fc410f-af3c-4e6c-80c9-4f526d991a93" containerName="registry-server"
Jul 17 20:48:51.738334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.738271    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fc410f-af3c-4e6c-80c9-4f526d991a93" containerName="registry-server"
Jul 17 20:48:51.738334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.738282    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d87fc080-0d95-4a1e-ae31-7d4488c5e174" containerName="revoke-keys"
Jul 17 20:48:51.738334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.738289    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e96e2a1-7cda-4615-bcb3-3eb42e3c4a68" containerName="revoke-keys"
Jul 17 20:48:51.741313 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.741295    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"
Jul 17 20:48:51.747542 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.747521    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"]
Jul 17 20:48:51.871108 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.871072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5635fe3-a166-45a5-a91f-b50382500a54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-hwzjv\" (UID: \"e5635fe3-a166-45a5-a91f-b50382500a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"
Jul 17 20:48:51.971983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.971950    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5635fe3-a166-45a5-a91f-b50382500a54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-hwzjv\" (UID: \"e5635fe3-a166-45a5-a91f-b50382500a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"
Jul 17 20:48:51.972584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:51.972563    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5635fe3-a166-45a5-a91f-b50382500a54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-hwzjv\" (UID: \"e5635fe3-a166-45a5-a91f-b50382500a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"
Jul 17 20:48:52.053167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:52.053085    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"
Jul 17 20:48:52.182764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:52.182739    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"]
Jul 17 20:48:52.184485 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:48:52.184457    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5635fe3_a166_45a5_a91f_b50382500a54.slice/crio-0e422f5ff424c9f0956faa5f86a0f1b4309453ff5845773c07eb28a701987ec1 WatchSource:0}: Error finding container 0e422f5ff424c9f0956faa5f86a0f1b4309453ff5845773c07eb28a701987ec1: Status 404 returned error can't find the container with id 0e422f5ff424c9f0956faa5f86a0f1b4309453ff5845773c07eb28a701987ec1
Jul 17 20:48:52.401625 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:52.401537    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" event={"ID":"e5635fe3-a166-45a5-a91f-b50382500a54","Type":"ContainerStarted","Data":"0e422f5ff424c9f0956faa5f86a0f1b4309453ff5845773c07eb28a701987ec1"}
Jul 17 20:48:53.407399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:53.407365    2572 generic.go:356] "Generic (PLEG): container finished" podID="e5635fe3-a166-45a5-a91f-b50382500a54" containerID="38fdaff150f0a09ffa99cf1436efd3310fafda19688c7aca0b51e046a070e0e6" exitCode=6
Jul 17 20:48:53.407776 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:53.407418    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" event={"ID":"e5635fe3-a166-45a5-a91f-b50382500a54","Type":"ContainerDied","Data":"38fdaff150f0a09ffa99cf1436efd3310fafda19688c7aca0b51e046a070e0e6"}
Jul 17 20:48:53.407776 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:53.407646    2572 scope.go:117] "RemoveContainer" containerID="38fdaff150f0a09ffa99cf1436efd3310fafda19688c7aca0b51e046a070e0e6"
Jul 17 20:48:54.413380 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:54.413345    2572 generic.go:356] "Generic (PLEG): container finished" podID="e5635fe3-a166-45a5-a91f-b50382500a54" containerID="9fe8906e5fedca78c7a19e0354247f6a6a5ce2e6469ee17e19e0897ad4e5e12c" exitCode=6
Jul 17 20:48:54.413787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:54.413434    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" event={"ID":"e5635fe3-a166-45a5-a91f-b50382500a54","Type":"ContainerDied","Data":"9fe8906e5fedca78c7a19e0354247f6a6a5ce2e6469ee17e19e0897ad4e5e12c"}
Jul 17 20:48:54.413787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:54.413493    2572 scope.go:117] "RemoveContainer" containerID="38fdaff150f0a09ffa99cf1436efd3310fafda19688c7aca0b51e046a070e0e6"
Jul 17 20:48:54.413787 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:54.413670    2572 scope.go:117] "RemoveContainer" containerID="9fe8906e5fedca78c7a19e0354247f6a6a5ce2e6469ee17e19e0897ad4e5e12c"
Jul 17 20:48:54.413945 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:54.413879    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-hwzjv_odh-ai-gateway-infra(e5635fe3-a166-45a5-a91f-b50382500a54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" podUID="e5635fe3-a166-45a5-a91f-b50382500a54"
Jul 17 20:48:55.419366 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:55.419342    2572 scope.go:117] "RemoveContainer" containerID="9fe8906e5fedca78c7a19e0354247f6a6a5ce2e6469ee17e19e0897ad4e5e12c"
Jul 17 20:48:55.419742 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:55.419546    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-hwzjv_odh-ai-gateway-infra(e5635fe3-a166-45a5-a91f-b50382500a54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" podUID="e5635fe3-a166-45a5-a91f-b50382500a54"
Jul 17 20:48:55.850028 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:55.849994    2572 scope.go:117] "RemoveContainer" containerID="fb2fef0ff74b57b9255c4c18bc3b02d283dd833b53776ac2937b244abfb2d60d"
Jul 17 20:48:56.425536 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:56.425455    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" event={"ID":"10f5c0b6-42a5-4107-83c2-dd78f57c2098","Type":"ContainerStarted","Data":"a7dc65e7b07758aac5a92861f9c24f68f48e858fdd2db7a4d2a1a24aabed0897"}
Jul 17 20:48:56.442311 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:56.442266    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" podStartSLOduration=17.442249591 podStartE2EDuration="17.442249591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:48:56.440961431 +0000 UTC m=+3171.134937578" watchObservedRunningTime="2026-07-17 20:48:56.442249591 +0000 UTC m=+3171.136225737"
Jul 17 20:48:57.432196 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:57.432165    2572 generic.go:356] "Generic (PLEG): container finished" podID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerID="a7dc65e7b07758aac5a92861f9c24f68f48e858fdd2db7a4d2a1a24aabed0897" exitCode=6
Jul 17 20:48:57.432687 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:57.432214    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" event={"ID":"10f5c0b6-42a5-4107-83c2-dd78f57c2098","Type":"ContainerDied","Data":"a7dc65e7b07758aac5a92861f9c24f68f48e858fdd2db7a4d2a1a24aabed0897"}
Jul 17 20:48:57.432687 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:57.432269    2572 scope.go:117] "RemoveContainer" containerID="fb2fef0ff74b57b9255c4c18bc3b02d283dd833b53776ac2937b244abfb2d60d"
Jul 17 20:48:57.432687 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:57.432558    2572 scope.go:117] "RemoveContainer" containerID="a7dc65e7b07758aac5a92861f9c24f68f48e858fdd2db7a4d2a1a24aabed0897"
Jul 17 20:48:57.432868 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:48:57.432767    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn_odh-ai-gateway-infra(10f5c0b6-42a5-4107-83c2-dd78f57c2098)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098"
Jul 17 20:48:57.468935 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:57.468895    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"]
Jul 17 20:48:58.571354 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:58.571332    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"
Jul 17 20:48:58.626770 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:58.626740    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f5c0b6-42a5-4107-83c2-dd78f57c2098-maas-api-service-ca\") pod \"10f5c0b6-42a5-4107-83c2-dd78f57c2098\" (UID: \"10f5c0b6-42a5-4107-83c2-dd78f57c2098\") "
Jul 17 20:48:58.627186 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:58.627161    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f5c0b6-42a5-4107-83c2-dd78f57c2098-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10f5c0b6-42a5-4107-83c2-dd78f57c2098" (UID: "10f5c0b6-42a5-4107-83c2-dd78f57c2098"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:48:58.727780 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:58.727739    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f5c0b6-42a5-4107-83c2-dd78f57c2098-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:48:59.446432 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:59.446404    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"
Jul 17 20:48:59.446432 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:59.446434    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn" event={"ID":"10f5c0b6-42a5-4107-83c2-dd78f57c2098","Type":"ContainerDied","Data":"c8fcbf32a3c251399f945bc1111b665c4d643db6602e25b9632d02a7374ccf4a"}
Jul 17 20:48:59.446672 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:59.446493    2572 scope.go:117] "RemoveContainer" containerID="a7dc65e7b07758aac5a92861f9c24f68f48e858fdd2db7a4d2a1a24aabed0897"
Jul 17 20:48:59.476132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:59.476106    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"]
Jul 17 20:48:59.479815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:59.479793    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-7nmfn"]
Jul 17 20:48:59.852984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:48:59.852951    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" path="/var/lib/kubelet/pods/10f5c0b6-42a5-4107-83c2-dd78f57c2098/volumes"
Jul 17 20:49:05.826863 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.826825    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"]
Jul 17 20:49:05.827348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827328    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.827348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827346    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.827477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827356    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.827477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827364    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.827477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827378    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.827477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827387    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.827477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827471    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.827724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.827483    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:05.832952 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.832931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"
Jul 17 20:49:05.837691 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.837659    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"]
Jul 17 20:49:05.986498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:05.986461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc26af-35e3-4b53-9c2d-81a98b95d338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-56mdn\" (UID: \"1cfc26af-35e3-4b53-9c2d-81a98b95d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"
Jul 17 20:49:06.087345 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:06.087250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc26af-35e3-4b53-9c2d-81a98b95d338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-56mdn\" (UID: \"1cfc26af-35e3-4b53-9c2d-81a98b95d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"
Jul 17 20:49:06.087841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:06.087821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc26af-35e3-4b53-9c2d-81a98b95d338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-56mdn\" (UID: \"1cfc26af-35e3-4b53-9c2d-81a98b95d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"
Jul 17 20:49:06.145662 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:06.145624    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"
Jul 17 20:49:06.277756 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:06.277734    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"]
Jul 17 20:49:06.279605 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:49:06.279578    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfc26af_35e3_4b53_9c2d_81a98b95d338.slice/crio-db815f5e5251e40e37ff93bd35733012a92428b17c9bf83b443e47f0a1cfce5a WatchSource:0}: Error finding container db815f5e5251e40e37ff93bd35733012a92428b17c9bf83b443e47f0a1cfce5a: Status 404 returned error can't find the container with id db815f5e5251e40e37ff93bd35733012a92428b17c9bf83b443e47f0a1cfce5a
Jul 17 20:49:06.481839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:06.481802    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" event={"ID":"1cfc26af-35e3-4b53-9c2d-81a98b95d338","Type":"ContainerStarted","Data":"db815f5e5251e40e37ff93bd35733012a92428b17c9bf83b443e47f0a1cfce5a"}
Jul 17 20:49:07.487816 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:07.487782    2572 generic.go:356] "Generic (PLEG): container finished" podID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerID="cca7d3358c84791f562b6971861190725da328d5b2cb93a44fd0f3a89e1b0f85" exitCode=6
Jul 17 20:49:07.488255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:07.487870    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" event={"ID":"1cfc26af-35e3-4b53-9c2d-81a98b95d338","Type":"ContainerDied","Data":"cca7d3358c84791f562b6971861190725da328d5b2cb93a44fd0f3a89e1b0f85"}
Jul 17 20:49:07.488255 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:07.488042    2572 scope.go:117] "RemoveContainer" containerID="cca7d3358c84791f562b6971861190725da328d5b2cb93a44fd0f3a89e1b0f85"
Jul 17 20:49:08.046987 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.046949    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"]
Jul 17 20:49:08.047348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.047335    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f5c0b6-42a5-4107-83c2-dd78f57c2098" containerName="revoke-keys"
Jul 17 20:49:08.050649 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.050633    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"
Jul 17 20:49:08.060667 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.060641    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"]
Jul 17 20:49:08.203374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.203329    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b261997-e07e-4ffa-b9ec-e475d93be65d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-blvxm\" (UID: \"7b261997-e07e-4ffa-b9ec-e475d93be65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"
Jul 17 20:49:08.304814 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.304731    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b261997-e07e-4ffa-b9ec-e475d93be65d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-blvxm\" (UID: \"7b261997-e07e-4ffa-b9ec-e475d93be65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"
Jul 17 20:49:08.305383 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.305362    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b261997-e07e-4ffa-b9ec-e475d93be65d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-blvxm\" (UID: \"7b261997-e07e-4ffa-b9ec-e475d93be65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"
Jul 17 20:49:08.362320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.362289    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"
Jul 17 20:49:08.488070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.488019    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"]
Jul 17 20:49:08.489751 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:49:08.489728    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b261997_e07e_4ffa_b9ec_e475d93be65d.slice/crio-dbba760dee74236df1c0f2012242dfb0c8f7293a1a86c994e447144e507ea6cc WatchSource:0}: Error finding container dbba760dee74236df1c0f2012242dfb0c8f7293a1a86c994e447144e507ea6cc: Status 404 returned error can't find the container with id dbba760dee74236df1c0f2012242dfb0c8f7293a1a86c994e447144e507ea6cc
Jul 17 20:49:08.494717 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.494694    2572 generic.go:356] "Generic (PLEG): container finished" podID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerID="bc7f84a5d312d9c5128303855adff1e291310ed15d084842417bfc5689393c83" exitCode=6
Jul 17 20:49:08.494801 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.494731    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" event={"ID":"1cfc26af-35e3-4b53-9c2d-81a98b95d338","Type":"ContainerDied","Data":"bc7f84a5d312d9c5128303855adff1e291310ed15d084842417bfc5689393c83"}
Jul 17 20:49:08.494801 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.494774    2572 scope.go:117] "RemoveContainer" containerID="cca7d3358c84791f562b6971861190725da328d5b2cb93a44fd0f3a89e1b0f85"
Jul 17 20:49:08.494984 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:08.494965    2572 scope.go:117] "RemoveContainer" containerID="bc7f84a5d312d9c5128303855adff1e291310ed15d084842417bfc5689393c83"
Jul 17 20:49:08.495234 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:08.495207    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-56mdn_odh-ai-gateway-infra(1cfc26af-35e3-4b53-9c2d-81a98b95d338)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338"
Jul 17 20:49:09.501119 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:09.501091    2572 scope.go:117] "RemoveContainer" containerID="bc7f84a5d312d9c5128303855adff1e291310ed15d084842417bfc5689393c83"
Jul 17 20:49:09.501572 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:09.501293    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-56mdn_odh-ai-gateway-infra(1cfc26af-35e3-4b53-9c2d-81a98b95d338)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338"
Jul 17 20:49:09.502299 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:09.502274    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerID="8845aef6ce829b6ff76a3e45f1b48d4dc370f79480248845ffef774328a5189c" exitCode=6
Jul 17 20:49:09.502407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:09.502343    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" event={"ID":"7b261997-e07e-4ffa-b9ec-e475d93be65d","Type":"ContainerDied","Data":"8845aef6ce829b6ff76a3e45f1b48d4dc370f79480248845ffef774328a5189c"}
Jul 17 20:49:09.502407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:09.502385    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" event={"ID":"7b261997-e07e-4ffa-b9ec-e475d93be65d","Type":"ContainerStarted","Data":"dbba760dee74236df1c0f2012242dfb0c8f7293a1a86c994e447144e507ea6cc"}
Jul 17 20:49:09.502506 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:09.502495    2572 scope.go:117] "RemoveContainer" containerID="8845aef6ce829b6ff76a3e45f1b48d4dc370f79480248845ffef774328a5189c"
Jul 17 20:49:09.848687 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:09.848658    2572 scope.go:117] "RemoveContainer" containerID="9fe8906e5fedca78c7a19e0354247f6a6a5ce2e6469ee17e19e0897ad4e5e12c"
Jul 17 20:49:10.508213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.508182    2572 generic.go:356] "Generic (PLEG): container finished" podID="e5635fe3-a166-45a5-a91f-b50382500a54" containerID="ad4a6cbe0077f97fe8e8b13c210fa56c3bf0076d6c0372084387c3b7f9d8f11d" exitCode=6
Jul 17 20:49:10.508619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.508257    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" event={"ID":"e5635fe3-a166-45a5-a91f-b50382500a54","Type":"ContainerDied","Data":"ad4a6cbe0077f97fe8e8b13c210fa56c3bf0076d6c0372084387c3b7f9d8f11d"}
Jul 17 20:49:10.508619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.508332    2572 scope.go:117] "RemoveContainer" containerID="9fe8906e5fedca78c7a19e0354247f6a6a5ce2e6469ee17e19e0897ad4e5e12c"
Jul 17 20:49:10.508723 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.508652    2572 scope.go:117] "RemoveContainer" containerID="ad4a6cbe0077f97fe8e8b13c210fa56c3bf0076d6c0372084387c3b7f9d8f11d"
Jul 17 20:49:10.508902 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:10.508880    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-hwzjv_odh-ai-gateway-infra(e5635fe3-a166-45a5-a91f-b50382500a54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" podUID="e5635fe3-a166-45a5-a91f-b50382500a54"
Jul 17 20:49:10.510096 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.510036    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerID="44036e538415792eb89307795f401258587a10583ccbaafc67e43b2c51b6f4ea" exitCode=6
Jul 17 20:49:10.510189 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.510136    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" event={"ID":"7b261997-e07e-4ffa-b9ec-e475d93be65d","Type":"ContainerDied","Data":"44036e538415792eb89307795f401258587a10583ccbaafc67e43b2c51b6f4ea"}
Jul 17 20:49:10.510320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.510308    2572 scope.go:117] "RemoveContainer" containerID="44036e538415792eb89307795f401258587a10583ccbaafc67e43b2c51b6f4ea"
Jul 17 20:49:10.510469 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:10.510454    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-blvxm_odh-ai-gateway-infra(7b261997-e07e-4ffa-b9ec-e475d93be65d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d"
Jul 17 20:49:10.518900 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:10.518885    2572 scope.go:117] "RemoveContainer" containerID="8845aef6ce829b6ff76a3e45f1b48d4dc370f79480248845ffef774328a5189c"
Jul 17 20:49:11.517223 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:11.517193    2572 scope.go:117] "RemoveContainer" containerID="44036e538415792eb89307795f401258587a10583ccbaafc67e43b2c51b6f4ea"
Jul 17 20:49:11.517661 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:11.517447    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-blvxm_odh-ai-gateway-infra(7b261997-e07e-4ffa-b9ec-e475d93be65d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d"
Jul 17 20:49:11.533700 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:11.533676    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"]
Jul 17 20:49:11.661506 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:11.661483    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"
Jul 17 20:49:11.733575 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:11.733544    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5635fe3-a166-45a5-a91f-b50382500a54-maas-api-service-ca\") pod \"e5635fe3-a166-45a5-a91f-b50382500a54\" (UID: \"e5635fe3-a166-45a5-a91f-b50382500a54\") "
Jul 17 20:49:11.733889 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:11.733864    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5635fe3-a166-45a5-a91f-b50382500a54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5635fe3-a166-45a5-a91f-b50382500a54" (UID: "e5635fe3-a166-45a5-a91f-b50382500a54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:49:11.835023 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:11.834932    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5635fe3-a166-45a5-a91f-b50382500a54-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:49:12.523207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:12.523177    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"
Jul 17 20:49:12.523677 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:12.523212    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv" event={"ID":"e5635fe3-a166-45a5-a91f-b50382500a54","Type":"ContainerDied","Data":"0e422f5ff424c9f0956faa5f86a0f1b4309453ff5845773c07eb28a701987ec1"}
Jul 17 20:49:12.523677 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:12.523269    2572 scope.go:117] "RemoveContainer" containerID="ad4a6cbe0077f97fe8e8b13c210fa56c3bf0076d6c0372084387c3b7f9d8f11d"
Jul 17 20:49:12.540109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:12.540057    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"]
Jul 17 20:49:12.546101 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:12.546082    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-hwzjv"]
Jul 17 20:49:13.853566 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:13.853529    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" path="/var/lib/kubelet/pods/e5635fe3-a166-45a5-a91f-b50382500a54/volumes"
Jul 17 20:49:18.794825 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:18.794788    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:49:22.848041 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:22.848007    2572 scope.go:117] "RemoveContainer" containerID="bc7f84a5d312d9c5128303855adff1e291310ed15d084842417bfc5689393c83"
Jul 17 20:49:23.588851 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:23.588819    2572 generic.go:356] "Generic (PLEG): container finished" podID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerID="95eb7bcecdd18d0fb56fd91cdc9a497a39acedebb7a9d68a269f6ea032c2bf50" exitCode=6
Jul 17 20:49:23.589066 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:23.588899    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" event={"ID":"1cfc26af-35e3-4b53-9c2d-81a98b95d338","Type":"ContainerDied","Data":"95eb7bcecdd18d0fb56fd91cdc9a497a39acedebb7a9d68a269f6ea032c2bf50"}
Jul 17 20:49:23.589066 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:23.588969    2572 scope.go:117] "RemoveContainer" containerID="bc7f84a5d312d9c5128303855adff1e291310ed15d084842417bfc5689393c83"
Jul 17 20:49:23.589315 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:23.589293    2572 scope.go:117] "RemoveContainer" containerID="95eb7bcecdd18d0fb56fd91cdc9a497a39acedebb7a9d68a269f6ea032c2bf50"
Jul 17 20:49:23.589495 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:23.589477    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-56mdn_odh-ai-gateway-infra(1cfc26af-35e3-4b53-9c2d-81a98b95d338)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338"
Jul 17 20:49:23.847902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:23.847870    2572 scope.go:117] "RemoveContainer" containerID="44036e538415792eb89307795f401258587a10583ccbaafc67e43b2c51b6f4ea"
Jul 17 20:49:24.597032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:24.596998    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" event={"ID":"7b261997-e07e-4ffa-b9ec-e475d93be65d","Type":"ContainerStarted","Data":"2f2a862c0b405b260003f870c5ce2c99951786098b4383856d974ee98a2c1aaa"}
Jul 17 20:49:24.617134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:24.617080    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" podStartSLOduration=16.617065286 podStartE2EDuration="16.617065286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:49:24.613220295 +0000 UTC m=+3199.307196453" watchObservedRunningTime="2026-07-17 20:49:24.617065286 +0000 UTC m=+3199.311041424"
Jul 17 20:49:24.617541 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:24.617527    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"]
Jul 17 20:49:24.757344 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:24.757319    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"
Jul 17 20:49:24.841711 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:24.841685    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc26af-35e3-4b53-9c2d-81a98b95d338-maas-api-service-ca\") pod \"1cfc26af-35e3-4b53-9c2d-81a98b95d338\" (UID: \"1cfc26af-35e3-4b53-9c2d-81a98b95d338\") "
Jul 17 20:49:24.842059 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:24.842023    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfc26af-35e3-4b53-9c2d-81a98b95d338-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1cfc26af-35e3-4b53-9c2d-81a98b95d338" (UID: "1cfc26af-35e3-4b53-9c2d-81a98b95d338"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:49:24.943072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:24.942962    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc26af-35e3-4b53-9c2d-81a98b95d338-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:49:25.603219 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.603188    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerID="2f2a862c0b405b260003f870c5ce2c99951786098b4383856d974ee98a2c1aaa" exitCode=6
Jul 17 20:49:25.603725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.603264    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" event={"ID":"7b261997-e07e-4ffa-b9ec-e475d93be65d","Type":"ContainerDied","Data":"2f2a862c0b405b260003f870c5ce2c99951786098b4383856d974ee98a2c1aaa"}
Jul 17 20:49:25.603725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.603339    2572 scope.go:117] "RemoveContainer" containerID="44036e538415792eb89307795f401258587a10583ccbaafc67e43b2c51b6f4ea"
Jul 17 20:49:25.603725 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.603663    2572 scope.go:117] "RemoveContainer" containerID="2f2a862c0b405b260003f870c5ce2c99951786098b4383856d974ee98a2c1aaa"
Jul 17 20:49:25.603912 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:25.603892    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-blvxm_odh-ai-gateway-infra(7b261997-e07e-4ffa-b9ec-e475d93be65d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d"
Jul 17 20:49:25.604646 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.604554    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"
Jul 17 20:49:25.604646 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.604571    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn" event={"ID":"1cfc26af-35e3-4b53-9c2d-81a98b95d338","Type":"ContainerDied","Data":"db815f5e5251e40e37ff93bd35733012a92428b17c9bf83b443e47f0a1cfce5a"}
Jul 17 20:49:25.614460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.614444    2572 scope.go:117] "RemoveContainer" containerID="95eb7bcecdd18d0fb56fd91cdc9a497a39acedebb7a9d68a269f6ea032c2bf50"
Jul 17 20:49:25.631851 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.631825    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"]
Jul 17 20:49:25.636526 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.636501    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"]
Jul 17 20:49:25.644699 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.644671    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-56mdn"]
Jul 17 20:49:25.855743 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:25.855661    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" path="/var/lib/kubelet/pods/1cfc26af-35e3-4b53-9c2d-81a98b95d338/volumes"
Jul 17 20:49:26.746697 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:26.746669    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"
Jul 17 20:49:26.773604 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:26.773574    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b261997-e07e-4ffa-b9ec-e475d93be65d-maas-api-service-ca\") pod \"7b261997-e07e-4ffa-b9ec-e475d93be65d\" (UID: \"7b261997-e07e-4ffa-b9ec-e475d93be65d\") "
Jul 17 20:49:26.774000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:26.773972    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b261997-e07e-4ffa-b9ec-e475d93be65d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b261997-e07e-4ffa-b9ec-e475d93be65d" (UID: "7b261997-e07e-4ffa-b9ec-e475d93be65d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:49:26.874262 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:26.874175    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b261997-e07e-4ffa-b9ec-e475d93be65d-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:49:27.618011 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:27.617979    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"
Jul 17 20:49:27.618216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:27.618021    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm" event={"ID":"7b261997-e07e-4ffa-b9ec-e475d93be65d","Type":"ContainerDied","Data":"dbba760dee74236df1c0f2012242dfb0c8f7293a1a86c994e447144e507ea6cc"}
Jul 17 20:49:27.618216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:27.618112    2572 scope.go:117] "RemoveContainer" containerID="2f2a862c0b405b260003f870c5ce2c99951786098b4383856d974ee98a2c1aaa"
Jul 17 20:49:27.639242 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:27.639207    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"]
Jul 17 20:49:27.643144 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:27.643119    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-blvxm"]
Jul 17 20:49:27.854210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:27.854180    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" path="/var/lib/kubelet/pods/7b261997-e07e-4ffa-b9ec-e475d93be65d/volumes"
Jul 17 20:49:29.865030 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865002    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"]
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865336    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865348    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865359    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865364    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865371    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865378    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865388    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865393    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865400    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865405    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:29.865414 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865416    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865421    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865467    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865474    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865481    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865488    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865493    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865499    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865504    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:29.865744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.865509    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:29.869815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.869800    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"
Jul 17 20:49:29.872222 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.872200    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:49:29.875438 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.875416    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"]
Jul 17 20:49:29.890924 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.890892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049e4c8-19a8-4941-891a-ae48640c87fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25\" (UID: \"b049e4c8-19a8-4941-891a-ae48640c87fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"
Jul 17 20:49:29.992290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.992254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049e4c8-19a8-4941-891a-ae48640c87fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25\" (UID: \"b049e4c8-19a8-4941-891a-ae48640c87fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"
Jul 17 20:49:29.992859 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:29.992838    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049e4c8-19a8-4941-891a-ae48640c87fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25\" (UID: \"b049e4c8-19a8-4941-891a-ae48640c87fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"
Jul 17 20:49:30.181118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:30.181013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"
Jul 17 20:49:30.314772 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:30.314746    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"]
Jul 17 20:49:30.316378 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:49:30.316341    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb049e4c8_19a8_4941_891a_ae48640c87fc.slice/crio-3218fd9100ee2ee55a71599efdce9c472c61cc50bd1033ca421f090fba01095d WatchSource:0}: Error finding container 3218fd9100ee2ee55a71599efdce9c472c61cc50bd1033ca421f090fba01095d: Status 404 returned error can't find the container with id 3218fd9100ee2ee55a71599efdce9c472c61cc50bd1033ca421f090fba01095d
Jul 17 20:49:30.637557 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:30.637529    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" event={"ID":"b049e4c8-19a8-4941-891a-ae48640c87fc","Type":"ContainerStarted","Data":"3218fd9100ee2ee55a71599efdce9c472c61cc50bd1033ca421f090fba01095d"}
Jul 17 20:49:31.643786 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:31.643751    2572 generic.go:356] "Generic (PLEG): container finished" podID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerID="931c63aafe16d143a03de01b6fcbd444a3df7a7113a752b351011caa63a66d57" exitCode=6
Jul 17 20:49:31.644190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:31.643804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" event={"ID":"b049e4c8-19a8-4941-891a-ae48640c87fc","Type":"ContainerDied","Data":"931c63aafe16d143a03de01b6fcbd444a3df7a7113a752b351011caa63a66d57"}
Jul 17 20:49:31.644190 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:31.643999    2572 scope.go:117] "RemoveContainer" containerID="931c63aafe16d143a03de01b6fcbd444a3df7a7113a752b351011caa63a66d57"
Jul 17 20:49:32.650668 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:32.650580    2572 generic.go:356] "Generic (PLEG): container finished" podID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerID="ddcb2922603d7b622f2c3e8f04b1708674b7913ae191bfd8197813d05caf9d45" exitCode=6
Jul 17 20:49:32.651137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:32.650666    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" event={"ID":"b049e4c8-19a8-4941-891a-ae48640c87fc","Type":"ContainerDied","Data":"ddcb2922603d7b622f2c3e8f04b1708674b7913ae191bfd8197813d05caf9d45"}
Jul 17 20:49:32.651137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:32.650722    2572 scope.go:117] "RemoveContainer" containerID="931c63aafe16d143a03de01b6fcbd444a3df7a7113a752b351011caa63a66d57"
Jul 17 20:49:32.651137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:32.650972    2572 scope.go:117] "RemoveContainer" containerID="ddcb2922603d7b622f2c3e8f04b1708674b7913ae191bfd8197813d05caf9d45"
Jul 17 20:49:32.651324 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:32.651210    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25_odh-ai-gateway-infra(b049e4c8-19a8-4941-891a-ae48640c87fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc"
Jul 17 20:49:33.656924 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:33.656893    2572 scope.go:117] "RemoveContainer" containerID="ddcb2922603d7b622f2c3e8f04b1708674b7913ae191bfd8197813d05caf9d45"
Jul 17 20:49:33.657329 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:33.657138    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25_odh-ai-gateway-infra(b049e4c8-19a8-4941-891a-ae48640c87fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc"
Jul 17 20:49:46.848806 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:46.848768    2572 scope.go:117] "RemoveContainer" containerID="ddcb2922603d7b622f2c3e8f04b1708674b7913ae191bfd8197813d05caf9d45"
Jul 17 20:49:47.722451 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:47.722417    2572 generic.go:356] "Generic (PLEG): container finished" podID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerID="9ed73737b2037d86c89954bb94e9bce152c83d27f01df7410466d74c08816241" exitCode=6
Jul 17 20:49:47.722652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:47.722488    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" event={"ID":"b049e4c8-19a8-4941-891a-ae48640c87fc","Type":"ContainerDied","Data":"9ed73737b2037d86c89954bb94e9bce152c83d27f01df7410466d74c08816241"}
Jul 17 20:49:47.722652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:47.722551    2572 scope.go:117] "RemoveContainer" containerID="ddcb2922603d7b622f2c3e8f04b1708674b7913ae191bfd8197813d05caf9d45"
Jul 17 20:49:47.722875 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:47.722859    2572 scope.go:117] "RemoveContainer" containerID="9ed73737b2037d86c89954bb94e9bce152c83d27f01df7410466d74c08816241"
Jul 17 20:49:47.723126 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:47.723103    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25_odh-ai-gateway-infra(b049e4c8-19a8-4941-891a-ae48640c87fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc"
Jul 17 20:49:48.748117 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:48.748085    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"]
Jul 17 20:49:48.881646 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:48.881622    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"
Jul 17 20:49:48.936072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:48.936019    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049e4c8-19a8-4941-891a-ae48640c87fc-maas-api-service-ca\") pod \"b049e4c8-19a8-4941-891a-ae48640c87fc\" (UID: \"b049e4c8-19a8-4941-891a-ae48640c87fc\") "
Jul 17 20:49:48.936419 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:48.936393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b049e4c8-19a8-4941-891a-ae48640c87fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b049e4c8-19a8-4941-891a-ae48640c87fc" (UID: "b049e4c8-19a8-4941-891a-ae48640c87fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:49:49.037076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.036982    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049e4c8-19a8-4941-891a-ae48640c87fc-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:49:49.734857 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.734821    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25" event={"ID":"b049e4c8-19a8-4941-891a-ae48640c87fc","Type":"ContainerDied","Data":"3218fd9100ee2ee55a71599efdce9c472c61cc50bd1033ca421f090fba01095d"}
Jul 17 20:49:49.735079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.734879    2572 scope.go:117] "RemoveContainer" containerID="9ed73737b2037d86c89954bb94e9bce152c83d27f01df7410466d74c08816241"
Jul 17 20:49:49.735079 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.734830    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"
Jul 17 20:49:49.765796 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.765768    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"]
Jul 17 20:49:49.777288 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.777264    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-dkg25"]
Jul 17 20:49:49.795403 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795369    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"]
Jul 17 20:49:49.795750 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795734    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795753    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795777    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795783    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cfc26af-35e3-4b53-9c2d-81a98b95d338" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795791    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795796    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5635fe3-a166-45a5-a91f-b50382500a54" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795809    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795817    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:49.795822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795826    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:49.796224 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795831    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:49.796224 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795883    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:49.796224 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795891    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:49.796224 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.795905    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b261997-e07e-4ffa-b9ec-e475d93be65d" containerName="revoke-keys"
Jul 17 20:49:49.799967 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.799948    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.802361 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.802313    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-327614\"/\"kube-root-ca.crt\""
Jul 17 20:49:49.802361 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.802326    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-327614\"/\"default-dockercfg-99jqs\""
Jul 17 20:49:49.802521 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.802336    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-327614\"/\"sub-test-model-327614-kserve-self-signed-certs\""
Jul 17 20:49:49.802521 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.802490    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-327614\"/\"openshift-service-ca.crt\""
Jul 17 20:49:49.809523 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.809499    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"]
Jul 17 20:49:49.845392 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.845361    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-dshm\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.845551 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.845419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-tmp-dir\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.845551 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.845502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ba8093-c1fe-47f4-a3f3-7212feac2236-tls-certs\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.845551 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.845535    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-model-cache\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.845692 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.845569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-home\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.845692 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.845650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/37ba8093-c1fe-47f4-a3f3-7212feac2236-kube-api-access-4zt67\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.853530 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.853501    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" path="/var/lib/kubelet/pods/b049e4c8-19a8-4941-891a-ae48640c87fc/volumes"
Jul 17 20:49:49.946833 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.946795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ba8093-c1fe-47f4-a3f3-7212feac2236-tls-certs\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.946833 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.946837    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-model-cache\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.947124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.946877    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-home\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.947124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.946905    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/37ba8093-c1fe-47f4-a3f3-7212feac2236-kube-api-access-4zt67\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.947124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.946931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-dshm\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.947124 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.947063    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-tmp-dir\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.947338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.947310    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-model-cache\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.947403 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.947372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-home\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.947528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.947423    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-tmp-dir\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.949164 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.949145    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-dshm\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.949478 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.949462    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ba8093-c1fe-47f4-a3f3-7212feac2236-tls-certs\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:49.956947 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:49.956928    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/37ba8093-c1fe-47f4-a3f3-7212feac2236-kube-api-access-4zt67\") pod \"sub-test-model-327614-kserve-bfcd4477c-jkw2q\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") " pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:50.108806 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:50.108724    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:49:50.242511 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:50.242488    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"]
Jul 17 20:49:50.245076 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:49:50.245030    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37ba8093_c1fe_47f4_a3f3_7212feac2236.slice/crio-d1e4aec17fe6c1bfd9a4469cb86d45f8b2bdef58941ea0bc5f335cc49425265f WatchSource:0}: Error finding container d1e4aec17fe6c1bfd9a4469cb86d45f8b2bdef58941ea0bc5f335cc49425265f: Status 404 returned error can't find the container with id d1e4aec17fe6c1bfd9a4469cb86d45f8b2bdef58941ea0bc5f335cc49425265f
Jul 17 20:49:50.742544 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:50.742503    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q" event={"ID":"37ba8093-c1fe-47f4-a3f3-7212feac2236","Type":"ContainerStarted","Data":"b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328"}
Jul 17 20:49:50.742775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:50.742557    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q" event={"ID":"37ba8093-c1fe-47f4-a3f3-7212feac2236","Type":"ContainerStarted","Data":"d1e4aec17fe6c1bfd9a4469cb86d45f8b2bdef58941ea0bc5f335cc49425265f"}
Jul 17 20:49:50.758779 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:50.758723    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q" podStartSLOduration=1.7587071399999998 podStartE2EDuration="1.75870714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:49:50.758263529 +0000 UTC m=+3225.452239676" watchObservedRunningTime="2026-07-17 20:49:50.75870714 +0000 UTC m=+3225.452683288"
Jul 17 20:49:51.788817 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.788782    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"]
Jul 17 20:49:51.789280 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.789267    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:51.789325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.789281    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:51.789377 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.789365    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b049e4c8-19a8-4941-891a-ae48640c87fc" containerName="revoke-keys"
Jul 17 20:49:51.792402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.792387    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"
Jul 17 20:49:51.794868 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.794850    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:49:51.798264 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.798239    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"]
Jul 17 20:49:51.863087 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.863028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fkhfw\" (UID: \"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"
Jul 17 20:49:51.963630 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.963594    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fkhfw\" (UID: \"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"
Jul 17 20:49:51.964341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:51.964323    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-fkhfw\" (UID: \"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"
Jul 17 20:49:52.103417 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:52.103318    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"
Jul 17 20:49:52.232746 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:52.232717    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"]
Jul 17 20:49:52.236354 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:49:52.236319    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac89eb8_c8f3_43cb_bdd6_baa0f5e2e4cb.slice/crio-b7f3f7b2c78eb27861b69a2ca403edd1c1f15eefff5fba3cbda581376f962b8f WatchSource:0}: Error finding container b7f3f7b2c78eb27861b69a2ca403edd1c1f15eefff5fba3cbda581376f962b8f: Status 404 returned error can't find the container with id b7f3f7b2c78eb27861b69a2ca403edd1c1f15eefff5fba3cbda581376f962b8f
Jul 17 20:49:52.754459 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:52.754424    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" event={"ID":"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb","Type":"ContainerStarted","Data":"ab2b24da2d94b05db92aa9542cb5cd466c11afd2cf18966fed3567bfa8a969a3"}
Jul 17 20:49:52.754677 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:52.754472    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" event={"ID":"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb","Type":"ContainerStarted","Data":"b7f3f7b2c78eb27861b69a2ca403edd1c1f15eefff5fba3cbda581376f962b8f"}
Jul 17 20:49:52.770484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:52.770440    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" podStartSLOduration=1.77042673 podStartE2EDuration="1.77042673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:49:52.768383022 +0000 UTC m=+3227.462359167" watchObservedRunningTime="2026-07-17 20:49:52.77042673 +0000 UTC m=+3227.464402874"
Jul 17 20:49:53.761084 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:53.761031    2572 generic.go:356] "Generic (PLEG): container finished" podID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerID="ab2b24da2d94b05db92aa9542cb5cd466c11afd2cf18966fed3567bfa8a969a3" exitCode=6
Jul 17 20:49:53.761525 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:53.761091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" event={"ID":"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb","Type":"ContainerDied","Data":"ab2b24da2d94b05db92aa9542cb5cd466c11afd2cf18966fed3567bfa8a969a3"}
Jul 17 20:49:53.761719 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:53.761696    2572 scope.go:117] "RemoveContainer" containerID="ab2b24da2d94b05db92aa9542cb5cd466c11afd2cf18966fed3567bfa8a969a3"
Jul 17 20:49:54.588752 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:54.588676    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:49:54.767020 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:54.766988    2572 generic.go:356] "Generic (PLEG): container finished" podID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerID="e0b85ac78b90922774790bf134f2e0da5309e1a355484cdd91d4dc48e540f6d6" exitCode=6
Jul 17 20:49:54.767509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:54.767078    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" event={"ID":"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb","Type":"ContainerDied","Data":"e0b85ac78b90922774790bf134f2e0da5309e1a355484cdd91d4dc48e540f6d6"}
Jul 17 20:49:54.767509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:54.767153    2572 scope.go:117] "RemoveContainer" containerID="ab2b24da2d94b05db92aa9542cb5cd466c11afd2cf18966fed3567bfa8a969a3"
Jul 17 20:49:54.767509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:54.767355    2572 scope.go:117] "RemoveContainer" containerID="e0b85ac78b90922774790bf134f2e0da5309e1a355484cdd91d4dc48e540f6d6"
Jul 17 20:49:54.767618 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:54.767579    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fkhfw_odh-ai-gateway-infra(5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb"
Jul 17 20:49:55.774005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:55.773974    2572 scope.go:117] "RemoveContainer" containerID="e0b85ac78b90922774790bf134f2e0da5309e1a355484cdd91d4dc48e540f6d6"
Jul 17 20:49:55.774396 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:55.774209    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fkhfw_odh-ai-gateway-infra(5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb"
Jul 17 20:49:55.867422 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:55.867388    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"]
Jul 17 20:49:55.871804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:55.871786    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"
Jul 17 20:49:55.877926 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:55.877872    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"]
Jul 17 20:49:55.993379 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:55.993338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d215fa-39e2-499a-8799-484566aadffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz\" (UID: \"73d215fa-39e2-499a-8799-484566aadffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"
Jul 17 20:49:56.094413 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:56.094323    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d215fa-39e2-499a-8799-484566aadffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz\" (UID: \"73d215fa-39e2-499a-8799-484566aadffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"
Jul 17 20:49:56.094907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:56.094888    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d215fa-39e2-499a-8799-484566aadffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz\" (UID: \"73d215fa-39e2-499a-8799-484566aadffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"
Jul 17 20:49:56.182713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:56.182682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"
Jul 17 20:49:56.312852 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:56.312830    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"]
Jul 17 20:49:56.314538 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:49:56.314510    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d215fa_39e2_499a_8799_484566aadffa.slice/crio-a99e5bef31fabaac859ffc8a676764221f3fc1865bf666577e660477a75226cc WatchSource:0}: Error finding container a99e5bef31fabaac859ffc8a676764221f3fc1865bf666577e660477a75226cc: Status 404 returned error can't find the container with id a99e5bef31fabaac859ffc8a676764221f3fc1865bf666577e660477a75226cc
Jul 17 20:49:56.780373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:56.780339    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" event={"ID":"73d215fa-39e2-499a-8799-484566aadffa","Type":"ContainerStarted","Data":"3187f14efb7baf08eaefc6aaa7253ef91080d27636d5318e935a3f8aa81613aa"}
Jul 17 20:49:56.780803 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:56.780392    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" event={"ID":"73d215fa-39e2-499a-8799-484566aadffa","Type":"ContainerStarted","Data":"a99e5bef31fabaac859ffc8a676764221f3fc1865bf666577e660477a75226cc"}
Jul 17 20:49:56.797199 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:56.797152    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" podStartSLOduration=1.797136925 podStartE2EDuration="1.797136925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:49:56.795023314 +0000 UTC m=+3231.488999459" watchObservedRunningTime="2026-07-17 20:49:56.797136925 +0000 UTC m=+3231.491113070"
Jul 17 20:49:57.786362 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:57.786328    2572 generic.go:356] "Generic (PLEG): container finished" podID="73d215fa-39e2-499a-8799-484566aadffa" containerID="3187f14efb7baf08eaefc6aaa7253ef91080d27636d5318e935a3f8aa81613aa" exitCode=6
Jul 17 20:49:57.786767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:57.786408    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" event={"ID":"73d215fa-39e2-499a-8799-484566aadffa","Type":"ContainerDied","Data":"3187f14efb7baf08eaefc6aaa7253ef91080d27636d5318e935a3f8aa81613aa"}
Jul 17 20:49:57.786767 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:57.786539    2572 scope.go:117] "RemoveContainer" containerID="3187f14efb7baf08eaefc6aaa7253ef91080d27636d5318e935a3f8aa81613aa"
Jul 17 20:49:58.086074 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.085982    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"]
Jul 17 20:49:58.089485 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.089467    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"
Jul 17 20:49:58.095895 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.095867    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"]
Jul 17 20:49:58.211749 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.211713    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5405b7c8-e81f-4c9f-b2f4-b4c162b6d629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-5594c\" (UID: \"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"
Jul 17 20:49:58.312368 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.312333    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5405b7c8-e81f-4c9f-b2f4-b4c162b6d629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-5594c\" (UID: \"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"
Jul 17 20:49:58.312908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.312888    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5405b7c8-e81f-4c9f-b2f4-b4c162b6d629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-5594c\" (UID: \"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"
Jul 17 20:49:58.400161 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.400076    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"
Jul 17 20:49:58.528435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.528396    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"]
Jul 17 20:49:58.530419 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:49:58.530394    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5405b7c8_e81f_4c9f_b2f4_b4c162b6d629.slice/crio-c6551964dcb6601f760189712e5059fa4f2004f6393192026ea590bb228dc127 WatchSource:0}: Error finding container c6551964dcb6601f760189712e5059fa4f2004f6393192026ea590bb228dc127: Status 404 returned error can't find the container with id c6551964dcb6601f760189712e5059fa4f2004f6393192026ea590bb228dc127
Jul 17 20:49:58.792455 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.792421    2572 generic.go:356] "Generic (PLEG): container finished" podID="73d215fa-39e2-499a-8799-484566aadffa" containerID="b24463915188e1745981d57acefbd26843f846e592bc490d996f2f66cff160bd" exitCode=6
Jul 17 20:49:58.792888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.792518    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" event={"ID":"73d215fa-39e2-499a-8799-484566aadffa","Type":"ContainerDied","Data":"b24463915188e1745981d57acefbd26843f846e592bc490d996f2f66cff160bd"}
Jul 17 20:49:58.792888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.792604    2572 scope.go:117] "RemoveContainer" containerID="3187f14efb7baf08eaefc6aaa7253ef91080d27636d5318e935a3f8aa81613aa"
Jul 17 20:49:58.792888 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.792715    2572 scope.go:117] "RemoveContainer" containerID="b24463915188e1745981d57acefbd26843f846e592bc490d996f2f66cff160bd"
Jul 17 20:49:58.793023 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:58.792927    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz_odh-ai-gateway-infra(73d215fa-39e2-499a-8799-484566aadffa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" podUID="73d215fa-39e2-499a-8799-484566aadffa"
Jul 17 20:49:58.793857 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:58.793838    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" event={"ID":"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629","Type":"ContainerStarted","Data":"c6551964dcb6601f760189712e5059fa4f2004f6393192026ea590bb228dc127"}
Jul 17 20:49:59.800939 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:59.800906    2572 generic.go:356] "Generic (PLEG): container finished" podID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerID="4a6056b9f4e80fcc6efac326b5b1e09c39eeb2b9e544f9f17e8889ff06ae1afb" exitCode=6
Jul 17 20:49:59.801371 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:59.800995    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" event={"ID":"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629","Type":"ContainerDied","Data":"4a6056b9f4e80fcc6efac326b5b1e09c39eeb2b9e544f9f17e8889ff06ae1afb"}
Jul 17 20:49:59.801371 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:59.801130    2572 scope.go:117] "RemoveContainer" containerID="4a6056b9f4e80fcc6efac326b5b1e09c39eeb2b9e544f9f17e8889ff06ae1afb"
Jul 17 20:49:59.802832 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:49:59.802815    2572 scope.go:117] "RemoveContainer" containerID="b24463915188e1745981d57acefbd26843f846e592bc490d996f2f66cff160bd"
Jul 17 20:49:59.802973 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:49:59.802957    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz_odh-ai-gateway-infra(73d215fa-39e2-499a-8799-484566aadffa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" podUID="73d215fa-39e2-499a-8799-484566aadffa"
Jul 17 20:50:00.109363 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.109285    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:50:00.109363 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.109331    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:50:00.121531 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.121506    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:50:00.809184 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.809096    2572 generic.go:356] "Generic (PLEG): container finished" podID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerID="4eab79c8fb1807489d5224ca89de092cc275fcd8fba5993ec3f8a3df398383ff" exitCode=6
Jul 17 20:50:00.809184 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.809172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" event={"ID":"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629","Type":"ContainerDied","Data":"4eab79c8fb1807489d5224ca89de092cc275fcd8fba5993ec3f8a3df398383ff"}
Jul 17 20:50:00.809713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.809224    2572 scope.go:117] "RemoveContainer" containerID="4a6056b9f4e80fcc6efac326b5b1e09c39eeb2b9e544f9f17e8889ff06ae1afb"
Jul 17 20:50:00.809713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.809428    2572 scope.go:117] "RemoveContainer" containerID="4eab79c8fb1807489d5224ca89de092cc275fcd8fba5993ec3f8a3df398383ff"
Jul 17 20:50:00.809713 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:00.809631    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-5594c_odh-ai-gateway-infra(5405b7c8-e81f-4c9f-b2f4-b4c162b6d629)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629"
Jul 17 20:50:00.820827 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:00.820811    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:50:01.815751 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:01.815722    2572 scope.go:117] "RemoveContainer" containerID="4eab79c8fb1807489d5224ca89de092cc275fcd8fba5993ec3f8a3df398383ff"
Jul 17 20:50:01.816165 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:01.815909    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-5594c_odh-ai-gateway-infra(5405b7c8-e81f-4c9f-b2f4-b4c162b6d629)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629"
Jul 17 20:50:09.853399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:09.853374    2572 scope.go:117] "RemoveContainer" containerID="e0b85ac78b90922774790bf134f2e0da5309e1a355484cdd91d4dc48e540f6d6"
Jul 17 20:50:10.848530 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:10.848499    2572 scope.go:117] "RemoveContainer" containerID="b24463915188e1745981d57acefbd26843f846e592bc490d996f2f66cff160bd"
Jul 17 20:50:10.860611 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:10.860582    2572 generic.go:356] "Generic (PLEG): container finished" podID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerID="d83d8f69ad688bc260c725d8b4fe3acdb0bf410e31de4c8407de636686b1ab4c" exitCode=6
Jul 17 20:50:10.860999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:10.860657    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" event={"ID":"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb","Type":"ContainerDied","Data":"d83d8f69ad688bc260c725d8b4fe3acdb0bf410e31de4c8407de636686b1ab4c"}
Jul 17 20:50:10.860999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:10.860717    2572 scope.go:117] "RemoveContainer" containerID="e0b85ac78b90922774790bf134f2e0da5309e1a355484cdd91d4dc48e540f6d6"
Jul 17 20:50:10.860999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:10.860986    2572 scope.go:117] "RemoveContainer" containerID="d83d8f69ad688bc260c725d8b4fe3acdb0bf410e31de4c8407de636686b1ab4c"
Jul 17 20:50:10.861218 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:10.861182    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-fkhfw_odh-ai-gateway-infra(5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb"
Jul 17 20:50:11.866442 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:11.866412    2572 generic.go:356] "Generic (PLEG): container finished" podID="73d215fa-39e2-499a-8799-484566aadffa" containerID="231912757a9296e48923ad03560cc8f95da99416363bcd636a91a11921409486" exitCode=6
Jul 17 20:50:11.866932 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:11.866478    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" event={"ID":"73d215fa-39e2-499a-8799-484566aadffa","Type":"ContainerDied","Data":"231912757a9296e48923ad03560cc8f95da99416363bcd636a91a11921409486"}
Jul 17 20:50:11.866932 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:11.866527    2572 scope.go:117] "RemoveContainer" containerID="b24463915188e1745981d57acefbd26843f846e592bc490d996f2f66cff160bd"
Jul 17 20:50:11.866932 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:11.866881    2572 scope.go:117] "RemoveContainer" containerID="231912757a9296e48923ad03560cc8f95da99416363bcd636a91a11921409486"
Jul 17 20:50:11.867153 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:11.867134    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz_odh-ai-gateway-infra(73d215fa-39e2-499a-8799-484566aadffa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" podUID="73d215fa-39e2-499a-8799-484566aadffa"
Jul 17 20:50:11.888203 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:11.888177    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"]
Jul 17 20:50:12.018471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.018447    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"
Jul 17 20:50:12.129340 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.129256    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb-maas-api-service-ca\") pod \"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb\" (UID: \"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb\") "
Jul 17 20:50:12.129689 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.129661    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" (UID: "5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:50:12.229982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.229945    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:12.875183 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.875156    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"
Jul 17 20:50:12.875575 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.875155    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw" event={"ID":"5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb","Type":"ContainerDied","Data":"b7f3f7b2c78eb27861b69a2ca403edd1c1f15eefff5fba3cbda581376f962b8f"}
Jul 17 20:50:12.875575 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.875284    2572 scope.go:117] "RemoveContainer" containerID="d83d8f69ad688bc260c725d8b4fe3acdb0bf410e31de4c8407de636686b1ab4c"
Jul 17 20:50:12.893019 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.892992    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"]
Jul 17 20:50:12.901220 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.901183    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"]
Jul 17 20:50:12.908717 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:12.908695    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-fkhfw"]
Jul 17 20:50:13.027640 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.027614    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"
Jul 17 20:50:13.137950 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.137861    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d215fa-39e2-499a-8799-484566aadffa-maas-api-service-ca\") pod \"73d215fa-39e2-499a-8799-484566aadffa\" (UID: \"73d215fa-39e2-499a-8799-484566aadffa\") "
Jul 17 20:50:13.138237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.138216    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d215fa-39e2-499a-8799-484566aadffa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73d215fa-39e2-499a-8799-484566aadffa" (UID: "73d215fa-39e2-499a-8799-484566aadffa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:50:13.238817 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.238784    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d215fa-39e2-499a-8799-484566aadffa-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:13.853063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.853028    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" path="/var/lib/kubelet/pods/5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb/volumes"
Jul 17 20:50:13.881207 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.881179    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"
Jul 17 20:50:13.881601 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.881181    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz" event={"ID":"73d215fa-39e2-499a-8799-484566aadffa","Type":"ContainerDied","Data":"a99e5bef31fabaac859ffc8a676764221f3fc1865bf666577e660477a75226cc"}
Jul 17 20:50:13.881601 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.881300    2572 scope.go:117] "RemoveContainer" containerID="231912757a9296e48923ad03560cc8f95da99416363bcd636a91a11921409486"
Jul 17 20:50:13.898858 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.898833    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"]
Jul 17 20:50:13.905467 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:13.905445    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-jdhgz"]
Jul 17 20:50:14.686001 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:14.685966    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:50:15.855040 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:15.855009    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d215fa-39e2-499a-8799-484566aadffa" path="/var/lib/kubelet/pods/73d215fa-39e2-499a-8799-484566aadffa/volumes"
Jul 17 20:50:16.847881 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:16.847852    2572 scope.go:117] "RemoveContainer" containerID="4eab79c8fb1807489d5224ca89de092cc275fcd8fba5993ec3f8a3df398383ff"
Jul 17 20:50:17.905894 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:17.905819    2572 generic.go:356] "Generic (PLEG): container finished" podID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerID="ab1b6bf97db15bb45df1f2e4b511b7f8039beda1be1d7adfbf45a624c797885b" exitCode=6
Jul 17 20:50:17.906390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:17.905889    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" event={"ID":"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629","Type":"ContainerDied","Data":"ab1b6bf97db15bb45df1f2e4b511b7f8039beda1be1d7adfbf45a624c797885b"}
Jul 17 20:50:17.906390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:17.905946    2572 scope.go:117] "RemoveContainer" containerID="4eab79c8fb1807489d5224ca89de092cc275fcd8fba5993ec3f8a3df398383ff"
Jul 17 20:50:17.906390 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:17.906258    2572 scope.go:117] "RemoveContainer" containerID="ab1b6bf97db15bb45df1f2e4b511b7f8039beda1be1d7adfbf45a624c797885b"
Jul 17 20:50:17.906565 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:17.906437    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-5594c_odh-ai-gateway-infra(5405b7c8-e81f-4c9f-b2f4-b4c162b6d629)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629"
Jul 17 20:50:18.932994 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:18.932945    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"]
Jul 17 20:50:19.067597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.067570    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"
Jul 17 20:50:19.188365 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.188285    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5405b7c8-e81f-4c9f-b2f4-b4c162b6d629-maas-api-service-ca\") pod \"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629\" (UID: \"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629\") "
Jul 17 20:50:19.188628 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.188606    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5405b7c8-e81f-4c9f-b2f4-b4c162b6d629-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" (UID: "5405b7c8-e81f-4c9f-b2f4-b4c162b6d629"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:50:19.289850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.289807    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5405b7c8-e81f-4c9f-b2f4-b4c162b6d629-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:19.909389 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909355    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"]
Jul 17 20:50:19.909684 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909672    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:19.909731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909685    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:19.909731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909696    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909702    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909714    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909720    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909726    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909731    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909744    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909748    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909754    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909759    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909765    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909770    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909776    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909783    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909828    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909835    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909841    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909846    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909851    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:19.909958 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.909857    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:19.912974 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.912958    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"
Jul 17 20:50:19.919971 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.919944    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"]
Jul 17 20:50:19.920435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.920409    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"
Jul 17 20:50:19.920549 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.920410    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c" event={"ID":"5405b7c8-e81f-4c9f-b2f4-b4c162b6d629","Type":"ContainerDied","Data":"c6551964dcb6601f760189712e5059fa4f2004f6393192026ea590bb228dc127"}
Jul 17 20:50:19.920622 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.920564    2572 scope.go:117] "RemoveContainer" containerID="ab1b6bf97db15bb45df1f2e4b511b7f8039beda1be1d7adfbf45a624c797885b"
Jul 17 20:50:19.940018 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.939991    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"]
Jul 17 20:50:19.945773 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.945749    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-5594c"]
Jul 17 20:50:19.996339 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:19.996308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186fc6f-b8d0-4f89-892c-ee7b106d5011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5\" (UID: \"f186fc6f-b8d0-4f89-892c-ee7b106d5011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"
Jul 17 20:50:20.096687 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:20.096658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186fc6f-b8d0-4f89-892c-ee7b106d5011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5\" (UID: \"f186fc6f-b8d0-4f89-892c-ee7b106d5011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"
Jul 17 20:50:20.097341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:20.097318    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186fc6f-b8d0-4f89-892c-ee7b106d5011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5\" (UID: \"f186fc6f-b8d0-4f89-892c-ee7b106d5011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"
Jul 17 20:50:20.227812 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:20.227778    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"
Jul 17 20:50:20.369819 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:20.369791    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"]
Jul 17 20:50:20.372737 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:50:20.372712    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186fc6f_b8d0_4f89_892c_ee7b106d5011.slice/crio-92f4d1153d8a11a64c85df76a8af3a74cfceb67bf0c618ee58f3336c8932bd6e WatchSource:0}: Error finding container 92f4d1153d8a11a64c85df76a8af3a74cfceb67bf0c618ee58f3336c8932bd6e: Status 404 returned error can't find the container with id 92f4d1153d8a11a64c85df76a8af3a74cfceb67bf0c618ee58f3336c8932bd6e
Jul 17 20:50:20.928421 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:20.928293    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" event={"ID":"f186fc6f-b8d0-4f89-892c-ee7b106d5011","Type":"ContainerStarted","Data":"d479f57f88008cba8a089ada05e3e21b681309353391dbcf5dcbabb8735f9b86"}
Jul 17 20:50:20.928421 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:20.928351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" event={"ID":"f186fc6f-b8d0-4f89-892c-ee7b106d5011","Type":"ContainerStarted","Data":"92f4d1153d8a11a64c85df76a8af3a74cfceb67bf0c618ee58f3336c8932bd6e"}
Jul 17 20:50:20.946887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:20.946839    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" podStartSLOduration=1.9468220939999998 podStartE2EDuration="1.946822094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:50:20.945381251 +0000 UTC m=+3255.639357395" watchObservedRunningTime="2026-07-17 20:50:20.946822094 +0000 UTC m=+3255.640798217"
Jul 17 20:50:21.852777 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:21.852736    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" path="/var/lib/kubelet/pods/5405b7c8-e81f-4c9f-b2f4-b4c162b6d629/volumes"
Jul 17 20:50:21.935476 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:21.935443    2572 generic.go:356] "Generic (PLEG): container finished" podID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerID="d479f57f88008cba8a089ada05e3e21b681309353391dbcf5dcbabb8735f9b86" exitCode=6
Jul 17 20:50:21.935651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:21.935486    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" event={"ID":"f186fc6f-b8d0-4f89-892c-ee7b106d5011","Type":"ContainerDied","Data":"d479f57f88008cba8a089ada05e3e21b681309353391dbcf5dcbabb8735f9b86"}
Jul 17 20:50:21.935738 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:21.935725    2572 scope.go:117] "RemoveContainer" containerID="d479f57f88008cba8a089ada05e3e21b681309353391dbcf5dcbabb8735f9b86"
Jul 17 20:50:22.942285 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:22.942194    2572 generic.go:356] "Generic (PLEG): container finished" podID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerID="f3107bf2fa700bbefc5e90dd4db438fef50af3a67eb87ab370f5d34d8656ef61" exitCode=6
Jul 17 20:50:22.942760 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:22.942281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" event={"ID":"f186fc6f-b8d0-4f89-892c-ee7b106d5011","Type":"ContainerDied","Data":"f3107bf2fa700bbefc5e90dd4db438fef50af3a67eb87ab370f5d34d8656ef61"}
Jul 17 20:50:22.942760 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:22.942381    2572 scope.go:117] "RemoveContainer" containerID="d479f57f88008cba8a089ada05e3e21b681309353391dbcf5dcbabb8735f9b86"
Jul 17 20:50:22.942760 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:22.942544    2572 scope.go:117] "RemoveContainer" containerID="f3107bf2fa700bbefc5e90dd4db438fef50af3a67eb87ab370f5d34d8656ef61"
Jul 17 20:50:22.942929 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:22.942773    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5_odh-ai-gateway-infra(f186fc6f-b8d0-4f89-892c-ee7b106d5011)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011"
Jul 17 20:50:23.948953 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:23.948921    2572 scope.go:117] "RemoveContainer" containerID="f3107bf2fa700bbefc5e90dd4db438fef50af3a67eb87ab370f5d34d8656ef61"
Jul 17 20:50:23.949419 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:23.949151    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5_odh-ai-gateway-infra(f186fc6f-b8d0-4f89-892c-ee7b106d5011)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011"
Jul 17 20:50:31.488386 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:31.488351    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"]
Jul 17 20:50:34.848256 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:34.848225    2572 scope.go:117] "RemoveContainer" containerID="f3107bf2fa700bbefc5e90dd4db438fef50af3a67eb87ab370f5d34d8656ef61"
Jul 17 20:50:35.587490 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:35.587453    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:50:36.008709 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:36.008677    2572 generic.go:356] "Generic (PLEG): container finished" podID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerID="f53fcae162af098e7672c6cb7a957ebbf090a2c32273050d3ba3ac08aef0bf53" exitCode=6
Jul 17 20:50:36.009135 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:36.008747    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" event={"ID":"f186fc6f-b8d0-4f89-892c-ee7b106d5011","Type":"ContainerDied","Data":"f53fcae162af098e7672c6cb7a957ebbf090a2c32273050d3ba3ac08aef0bf53"}
Jul 17 20:50:36.009135 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:36.008803    2572 scope.go:117] "RemoveContainer" containerID="f3107bf2fa700bbefc5e90dd4db438fef50af3a67eb87ab370f5d34d8656ef61"
Jul 17 20:50:36.009135 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:36.009082    2572 scope.go:117] "RemoveContainer" containerID="f53fcae162af098e7672c6cb7a957ebbf090a2c32273050d3ba3ac08aef0bf53"
Jul 17 20:50:36.009288 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:36.009267    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5_odh-ai-gateway-infra(f186fc6f-b8d0-4f89-892c-ee7b106d5011)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011"
Jul 17 20:50:37.040406 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:37.040377    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"]
Jul 17 20:50:37.181373 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:37.181351    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"
Jul 17 20:50:37.233603 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:37.233569    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186fc6f-b8d0-4f89-892c-ee7b106d5011-maas-api-service-ca\") pod \"f186fc6f-b8d0-4f89-892c-ee7b106d5011\" (UID: \"f186fc6f-b8d0-4f89-892c-ee7b106d5011\") "
Jul 17 20:50:37.233902 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:37.233881    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f186fc6f-b8d0-4f89-892c-ee7b106d5011-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f186fc6f-b8d0-4f89-892c-ee7b106d5011" (UID: "f186fc6f-b8d0-4f89-892c-ee7b106d5011"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:50:37.334249 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:37.334168    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186fc6f-b8d0-4f89-892c-ee7b106d5011-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:38.020663 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:38.020630    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"
Jul 17 20:50:38.020663 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:38.020661    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5" event={"ID":"f186fc6f-b8d0-4f89-892c-ee7b106d5011","Type":"ContainerDied","Data":"92f4d1153d8a11a64c85df76a8af3a74cfceb67bf0c618ee58f3336c8932bd6e"}
Jul 17 20:50:38.020912 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:38.020711    2572 scope.go:117] "RemoveContainer" containerID="f53fcae162af098e7672c6cb7a957ebbf090a2c32273050d3ba3ac08aef0bf53"
Jul 17 20:50:38.036869 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:38.036848    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"]
Jul 17 20:50:38.042886 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:38.042859    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-w2wf5"]
Jul 17 20:50:39.852965 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:39.852930    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" path="/var/lib/kubelet/pods/f186fc6f-b8d0-4f89-892c-ee7b106d5011/volumes"
Jul 17 20:50:45.939271 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939236    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"]
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939538    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939549    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939562    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939567    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939575    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939580    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939587    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939592    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939643    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939650    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939650 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939655    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f186fc6f-b8d0-4f89-892c-ee7b106d5011" containerName="revoke-keys"
Jul 17 20:50:45.939997 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939662    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d215fa-39e2-499a-8799-484566aadffa" containerName="revoke-keys"
Jul 17 20:50:45.939997 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939669    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac89eb8-c8f3-43cb-bdd6-baa0f5e2e4cb" containerName="revoke-keys"
Jul 17 20:50:45.939997 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.939675    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5405b7c8-e81f-4c9f-b2f4-b4c162b6d629" containerName="revoke-keys"
Jul 17 20:50:45.942714 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.942698    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"
Jul 17 20:50:45.946254 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.946235    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:50:45.963577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:45.963552    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"]
Jul 17 20:50:46.005841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.005813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c243ad-338e-4f7f-a75e-e34b5664585a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp\" (UID: \"08c243ad-338e-4f7f-a75e-e34b5664585a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"
Jul 17 20:50:46.107163 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.107134    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c243ad-338e-4f7f-a75e-e34b5664585a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp\" (UID: \"08c243ad-338e-4f7f-a75e-e34b5664585a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"
Jul 17 20:50:46.107738 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.107720    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c243ad-338e-4f7f-a75e-e34b5664585a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp\" (UID: \"08c243ad-338e-4f7f-a75e-e34b5664585a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"
Jul 17 20:50:46.252892 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.252859    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"
Jul 17 20:50:46.385663 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.385635    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"]
Jul 17 20:50:46.387380 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:50:46.387357    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c243ad_338e_4f7f_a75e_e34b5664585a.slice/crio-408c48b502abba77de37c36935ce0b5d85002d9325543bf62feeb0e8d48bddd7 WatchSource:0}: Error finding container 408c48b502abba77de37c36935ce0b5d85002d9325543bf62feeb0e8d48bddd7: Status 404 returned error can't find the container with id 408c48b502abba77de37c36935ce0b5d85002d9325543bf62feeb0e8d48bddd7
Jul 17 20:50:46.508260 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.508144    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q" podUID="37ba8093-c1fe-47f4-a3f3-7212feac2236" containerName="main" containerID="cri-o://b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328" gracePeriod=45
Jul 17 20:50:46.746823 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.746798    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:50:46.813107 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813003    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-dshm\") pod \"37ba8093-c1fe-47f4-a3f3-7212feac2236\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") "
Jul 17 20:50:46.813107 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813074    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-home\") pod \"37ba8093-c1fe-47f4-a3f3-7212feac2236\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") "
Jul 17 20:50:46.813323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813118    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/37ba8093-c1fe-47f4-a3f3-7212feac2236-kube-api-access-4zt67\") pod \"37ba8093-c1fe-47f4-a3f3-7212feac2236\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") "
Jul 17 20:50:46.813323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813145    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-model-cache\") pod \"37ba8093-c1fe-47f4-a3f3-7212feac2236\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") "
Jul 17 20:50:46.813323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813174    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ba8093-c1fe-47f4-a3f3-7212feac2236-tls-certs\") pod \"37ba8093-c1fe-47f4-a3f3-7212feac2236\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") "
Jul 17 20:50:46.813323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813197    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-tmp-dir\") pod \"37ba8093-c1fe-47f4-a3f3-7212feac2236\" (UID: \"37ba8093-c1fe-47f4-a3f3-7212feac2236\") "
Jul 17 20:50:46.813323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813282    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-home" (OuterVolumeSpecName: "home") pod "37ba8093-c1fe-47f4-a3f3-7212feac2236" (UID: "37ba8093-c1fe-47f4-a3f3-7212feac2236"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:50:46.813577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813420    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-home\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:46.813577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813415    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-model-cache" (OuterVolumeSpecName: "model-cache") pod "37ba8093-c1fe-47f4-a3f3-7212feac2236" (UID: "37ba8093-c1fe-47f4-a3f3-7212feac2236"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:50:46.813577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.813481    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37ba8093-c1fe-47f4-a3f3-7212feac2236" (UID: "37ba8093-c1fe-47f4-a3f3-7212feac2236"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:50:46.815290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.815264    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ba8093-c1fe-47f4-a3f3-7212feac2236-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37ba8093-c1fe-47f4-a3f3-7212feac2236" (UID: "37ba8093-c1fe-47f4-a3f3-7212feac2236"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 20:50:46.815380 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.815272    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-dshm" (OuterVolumeSpecName: "dshm") pod "37ba8093-c1fe-47f4-a3f3-7212feac2236" (UID: "37ba8093-c1fe-47f4-a3f3-7212feac2236"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 20:50:46.815380 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.815317    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ba8093-c1fe-47f4-a3f3-7212feac2236-kube-api-access-4zt67" (OuterVolumeSpecName: "kube-api-access-4zt67") pod "37ba8093-c1fe-47f4-a3f3-7212feac2236" (UID: "37ba8093-c1fe-47f4-a3f3-7212feac2236"). InnerVolumeSpecName "kube-api-access-4zt67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:50:46.914357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.914313    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/37ba8093-c1fe-47f4-a3f3-7212feac2236-kube-api-access-4zt67\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:46.914357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.914354    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-model-cache\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:46.914357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.914367    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ba8093-c1fe-47f4-a3f3-7212feac2236-tls-certs\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:46.914688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.914379    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-tmp-dir\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:46.914688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:46.914390    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ba8093-c1fe-47f4-a3f3-7212feac2236-dshm\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:50:47.066460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.066375    2572 generic.go:356] "Generic (PLEG): container finished" podID="37ba8093-c1fe-47f4-a3f3-7212feac2236" containerID="b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328" exitCode=0
Jul 17 20:50:47.066460 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.066418    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"
Jul 17 20:50:47.066954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.066467    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q" event={"ID":"37ba8093-c1fe-47f4-a3f3-7212feac2236","Type":"ContainerDied","Data":"b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328"}
Jul 17 20:50:47.066954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.066516    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q" event={"ID":"37ba8093-c1fe-47f4-a3f3-7212feac2236","Type":"ContainerDied","Data":"d1e4aec17fe6c1bfd9a4469cb86d45f8b2bdef58941ea0bc5f335cc49425265f"}
Jul 17 20:50:47.066954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.066569    2572 scope.go:117] "RemoveContainer" containerID="b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328"
Jul 17 20:50:47.068160 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.068134    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" event={"ID":"08c243ad-338e-4f7f-a75e-e34b5664585a","Type":"ContainerStarted","Data":"a45c81f59f915d2b27423eda54e96f2574caa6285d14685a8e413c0e38ee29fe"}
Jul 17 20:50:47.068270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.068189    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" event={"ID":"08c243ad-338e-4f7f-a75e-e34b5664585a","Type":"ContainerStarted","Data":"408c48b502abba77de37c36935ce0b5d85002d9325543bf62feeb0e8d48bddd7"}
Jul 17 20:50:47.075212 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.075197    2572 scope.go:117] "RemoveContainer" containerID="b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328"
Jul 17 20:50:47.075451 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:47.075435    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328\": container with ID starting with b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328 not found: ID does not exist" containerID="b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328"
Jul 17 20:50:47.075490 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.075459    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328"} err="failed to get container status \"b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328\": rpc error: code = NotFound desc = could not find container \"b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328\": container with ID starting with b10d7011f1d4fb37d99542df69323ac4542faa05e06090553606c6bb0a12c328 not found: ID does not exist"
Jul 17 20:50:47.096960 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.096920    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" podStartSLOduration=2.096905245 podStartE2EDuration="2.096905245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:50:47.09532825 +0000 UTC m=+3281.789304397" watchObservedRunningTime="2026-07-17 20:50:47.096905245 +0000 UTC m=+3281.790881390"
Jul 17 20:50:47.111707 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.111684    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"]
Jul 17 20:50:47.119332 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.119310    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-327614/sub-test-model-327614-kserve-bfcd4477c-jkw2q"]
Jul 17 20:50:47.853716 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:47.853683    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ba8093-c1fe-47f4-a3f3-7212feac2236" path="/var/lib/kubelet/pods/37ba8093-c1fe-47f4-a3f3-7212feac2236/volumes"
Jul 17 20:50:48.074217 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:48.074181    2572 generic.go:356] "Generic (PLEG): container finished" podID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerID="a45c81f59f915d2b27423eda54e96f2574caa6285d14685a8e413c0e38ee29fe" exitCode=6
Jul 17 20:50:48.074713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:48.074265    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" event={"ID":"08c243ad-338e-4f7f-a75e-e34b5664585a","Type":"ContainerDied","Data":"a45c81f59f915d2b27423eda54e96f2574caa6285d14685a8e413c0e38ee29fe"}
Jul 17 20:50:48.074713 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:48.074470    2572 scope.go:117] "RemoveContainer" containerID="a45c81f59f915d2b27423eda54e96f2574caa6285d14685a8e413c0e38ee29fe"
Jul 17 20:50:49.081979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:49.081895    2572 generic.go:356] "Generic (PLEG): container finished" podID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerID="16bd77f9e03784806bbeae4c30c928ea9a271687a255e53c1c6617f95e2b5fbf" exitCode=6
Jul 17 20:50:49.082503 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:49.081989    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" event={"ID":"08c243ad-338e-4f7f-a75e-e34b5664585a","Type":"ContainerDied","Data":"16bd77f9e03784806bbeae4c30c928ea9a271687a255e53c1c6617f95e2b5fbf"}
Jul 17 20:50:49.082503 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:49.082068    2572 scope.go:117] "RemoveContainer" containerID="a45c81f59f915d2b27423eda54e96f2574caa6285d14685a8e413c0e38ee29fe"
Jul 17 20:50:49.082503 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:49.082191    2572 scope.go:117] "RemoveContainer" containerID="16bd77f9e03784806bbeae4c30c928ea9a271687a255e53c1c6617f95e2b5fbf"
Jul 17 20:50:49.082503 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:49.082398    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp_odh-ai-gateway-infra(08c243ad-338e-4f7f-a75e-e34b5664585a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a"
Jul 17 20:50:50.088641 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:50.088614    2572 scope.go:117] "RemoveContainer" containerID="16bd77f9e03784806bbeae4c30c928ea9a271687a255e53c1c6617f95e2b5fbf"
Jul 17 20:50:50.089017 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:50.088766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp_odh-ai-gateway-infra(08c243ad-338e-4f7f-a75e-e34b5664585a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a"
Jul 17 20:50:51.840783 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:51.840750    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"]
Jul 17 20:50:51.841176 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:51.841102    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ba8093-c1fe-47f4-a3f3-7212feac2236" containerName="main"
Jul 17 20:50:51.841176 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:51.841114    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ba8093-c1fe-47f4-a3f3-7212feac2236" containerName="main"
Jul 17 20:50:51.841176 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:51.841173    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ba8093-c1fe-47f4-a3f3-7212feac2236" containerName="main"
Jul 17 20:50:51.845116 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:51.845101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"
Jul 17 20:50:51.855175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:51.855148    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"]
Jul 17 20:50:51.958722 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:51.958685    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-df4bb\" (UID: \"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"
Jul 17 20:50:52.060153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:52.060121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-df4bb\" (UID: \"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"
Jul 17 20:50:52.060730 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:52.060712    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-df4bb\" (UID: \"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"
Jul 17 20:50:52.156630 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:52.156545    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"
Jul 17 20:50:52.284752 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:52.284731    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"]
Jul 17 20:50:52.286506 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:50:52.286478    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9a17d6_8a0e_4ec9_a50f_c7a1344a802f.slice/crio-3843afaa4fa08713b982f9950b09c54ee3387216fb5877aa11d128b9de8eedfe WatchSource:0}: Error finding container 3843afaa4fa08713b982f9950b09c54ee3387216fb5877aa11d128b9de8eedfe: Status 404 returned error can't find the container with id 3843afaa4fa08713b982f9950b09c54ee3387216fb5877aa11d128b9de8eedfe
Jul 17 20:50:53.104503 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:53.104468    2572 generic.go:356] "Generic (PLEG): container finished" podID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerID="895beb5b049e69700afc9f5d27707610dc280d922f1095cb391e834d440c193a" exitCode=6
Jul 17 20:50:53.104884 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:53.104555    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" event={"ID":"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f","Type":"ContainerDied","Data":"895beb5b049e69700afc9f5d27707610dc280d922f1095cb391e834d440c193a"}
Jul 17 20:50:53.104884 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:53.104604    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" event={"ID":"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f","Type":"ContainerStarted","Data":"3843afaa4fa08713b982f9950b09c54ee3387216fb5877aa11d128b9de8eedfe"}
Jul 17 20:50:53.104884 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:53.104712    2572 scope.go:117] "RemoveContainer" containerID="895beb5b049e69700afc9f5d27707610dc280d922f1095cb391e834d440c193a"
Jul 17 20:50:54.110799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:54.110706    2572 generic.go:356] "Generic (PLEG): container finished" podID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerID="e9bbb4b4ea2f7aa4d22ece8c4bea135aaeff0e33a2aee500f79c6a923548aa99" exitCode=6
Jul 17 20:50:54.110799 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:54.110739    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" event={"ID":"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f","Type":"ContainerDied","Data":"e9bbb4b4ea2f7aa4d22ece8c4bea135aaeff0e33a2aee500f79c6a923548aa99"}
Jul 17 20:50:54.111319 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:54.110802    2572 scope.go:117] "RemoveContainer" containerID="895beb5b049e69700afc9f5d27707610dc280d922f1095cb391e834d440c193a"
Jul 17 20:50:54.111319 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:54.111033    2572 scope.go:117] "RemoveContainer" containerID="e9bbb4b4ea2f7aa4d22ece8c4bea135aaeff0e33a2aee500f79c6a923548aa99"
Jul 17 20:50:54.111319 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:54.111272    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-df4bb_odh-ai-gateway-infra(9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f"
Jul 17 20:50:55.117281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:55.117254    2572 scope.go:117] "RemoveContainer" containerID="e9bbb4b4ea2f7aa4d22ece8c4bea135aaeff0e33a2aee500f79c6a923548aa99"
Jul 17 20:50:55.117662 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:50:55.117418    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-df4bb_odh-ai-gateway-infra(9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f"
Jul 17 20:50:55.183081 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:55.183034    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:50:58.178087 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.178056    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"]
Jul 17 20:50:58.182683 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.182665    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"
Jul 17 20:50:58.205796 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.205769    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"]
Jul 17 20:50:58.313113 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.313075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aace9f7-4168-4f44-8031-060b8e85aef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w\" (UID: \"6aace9f7-4168-4f44-8031-060b8e85aef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"
Jul 17 20:50:58.413784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.413745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aace9f7-4168-4f44-8031-060b8e85aef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w\" (UID: \"6aace9f7-4168-4f44-8031-060b8e85aef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"
Jul 17 20:50:58.414397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.414378    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aace9f7-4168-4f44-8031-060b8e85aef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w\" (UID: \"6aace9f7-4168-4f44-8031-060b8e85aef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"
Jul 17 20:50:58.491897 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.491863    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"
Jul 17 20:50:58.627678 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:58.627646    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"]
Jul 17 20:50:58.629466 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:50:58.629436    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aace9f7_4168_4f44_8031_060b8e85aef8.slice/crio-9d3cfb684bcd61521fefb481660c245e981e89216aa37e456d7a1ae3245ba1df WatchSource:0}: Error finding container 9d3cfb684bcd61521fefb481660c245e981e89216aa37e456d7a1ae3245ba1df: Status 404 returned error can't find the container with id 9d3cfb684bcd61521fefb481660c245e981e89216aa37e456d7a1ae3245ba1df
Jul 17 20:50:59.139542 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:59.139463    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" event={"ID":"6aace9f7-4168-4f44-8031-060b8e85aef8","Type":"ContainerStarted","Data":"d3a0bd322c380f2e7c288e6cc3154fc761a1eda5ff325a2a0f926a5e6b5d88e1"}
Jul 17 20:50:59.139542 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:59.139511    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" event={"ID":"6aace9f7-4168-4f44-8031-060b8e85aef8","Type":"ContainerStarted","Data":"9d3cfb684bcd61521fefb481660c245e981e89216aa37e456d7a1ae3245ba1df"}
Jul 17 20:50:59.157688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:50:59.157644    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" podStartSLOduration=1.157628264 podStartE2EDuration="1.157628264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:50:59.156024272 +0000 UTC m=+3293.850000418" watchObservedRunningTime="2026-07-17 20:50:59.157628264 +0000 UTC m=+3293.851604409"
Jul 17 20:51:00.145447 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:00.145411    2572 generic.go:356] "Generic (PLEG): container finished" podID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerID="d3a0bd322c380f2e7c288e6cc3154fc761a1eda5ff325a2a0f926a5e6b5d88e1" exitCode=6
Jul 17 20:51:00.145821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:00.145492    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" event={"ID":"6aace9f7-4168-4f44-8031-060b8e85aef8","Type":"ContainerDied","Data":"d3a0bd322c380f2e7c288e6cc3154fc761a1eda5ff325a2a0f926a5e6b5d88e1"}
Jul 17 20:51:00.145821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:00.145681    2572 scope.go:117] "RemoveContainer" containerID="d3a0bd322c380f2e7c288e6cc3154fc761a1eda5ff325a2a0f926a5e6b5d88e1"
Jul 17 20:51:01.152137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:01.152026    2572 generic.go:356] "Generic (PLEG): container finished" podID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerID="27552100bfb1655085c2ca9820eaea3dc82922d5b7ef2558a90a5704c4758a81" exitCode=6
Jul 17 20:51:01.152137 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:01.152115    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" event={"ID":"6aace9f7-4168-4f44-8031-060b8e85aef8","Type":"ContainerDied","Data":"27552100bfb1655085c2ca9820eaea3dc82922d5b7ef2558a90a5704c4758a81"}
Jul 17 20:51:01.152630 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:01.152178    2572 scope.go:117] "RemoveContainer" containerID="d3a0bd322c380f2e7c288e6cc3154fc761a1eda5ff325a2a0f926a5e6b5d88e1"
Jul 17 20:51:01.152630 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:01.152344    2572 scope.go:117] "RemoveContainer" containerID="27552100bfb1655085c2ca9820eaea3dc82922d5b7ef2558a90a5704c4758a81"
Jul 17 20:51:01.152630 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:01.152541    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w_odh-ai-gateway-infra(6aace9f7-4168-4f44-8031-060b8e85aef8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8"
Jul 17 20:51:02.159250 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:02.159223    2572 scope.go:117] "RemoveContainer" containerID="27552100bfb1655085c2ca9820eaea3dc82922d5b7ef2558a90a5704c4758a81"
Jul 17 20:51:02.159627 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:02.159395    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w_odh-ai-gateway-infra(6aace9f7-4168-4f44-8031-060b8e85aef8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8"
Jul 17 20:51:04.847932 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:04.847906    2572 scope.go:117] "RemoveContainer" containerID="16bd77f9e03784806bbeae4c30c928ea9a271687a255e53c1c6617f95e2b5fbf"
Jul 17 20:51:05.177144 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:05.177065    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" event={"ID":"08c243ad-338e-4f7f-a75e-e34b5664585a","Type":"ContainerStarted","Data":"8510e929bbba7baaaf9728f6d7c1b115d443b71823c40a12adedd97eb2c354fd"}
Jul 17 20:51:06.184118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:06.184083    2572 generic.go:356] "Generic (PLEG): container finished" podID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerID="8510e929bbba7baaaf9728f6d7c1b115d443b71823c40a12adedd97eb2c354fd" exitCode=6
Jul 17 20:51:06.184584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:06.184159    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" event={"ID":"08c243ad-338e-4f7f-a75e-e34b5664585a","Type":"ContainerDied","Data":"8510e929bbba7baaaf9728f6d7c1b115d443b71823c40a12adedd97eb2c354fd"}
Jul 17 20:51:06.184584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:06.184231    2572 scope.go:117] "RemoveContainer" containerID="16bd77f9e03784806bbeae4c30c928ea9a271687a255e53c1c6617f95e2b5fbf"
Jul 17 20:51:06.184584 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:06.184458    2572 scope.go:117] "RemoveContainer" containerID="8510e929bbba7baaaf9728f6d7c1b115d443b71823c40a12adedd97eb2c354fd"
Jul 17 20:51:06.184729 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:06.184651    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp_odh-ai-gateway-infra(08c243ad-338e-4f7f-a75e-e34b5664585a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a"
Jul 17 20:51:06.205026 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:06.204998    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"]
Jul 17 20:51:06.413899 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:06.413867    2572 scope.go:117] "RemoveContainer" containerID="194456adff718cc9879c44ddbcbc42d1d1f42909512fe454b0f2a21abc169f0d"
Jul 17 20:51:07.320747 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:07.320727    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"
Jul 17 20:51:07.389949 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:07.389922    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c243ad-338e-4f7f-a75e-e34b5664585a-maas-api-service-ca\") pod \"08c243ad-338e-4f7f-a75e-e34b5664585a\" (UID: \"08c243ad-338e-4f7f-a75e-e34b5664585a\") "
Jul 17 20:51:07.390337 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:07.390313    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c243ad-338e-4f7f-a75e-e34b5664585a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08c243ad-338e-4f7f-a75e-e34b5664585a" (UID: "08c243ad-338e-4f7f-a75e-e34b5664585a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:51:07.490617 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:07.490571    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c243ad-338e-4f7f-a75e-e34b5664585a-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:51:07.853589 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:07.853511    2572 scope.go:117] "RemoveContainer" containerID="e9bbb4b4ea2f7aa4d22ece8c4bea135aaeff0e33a2aee500f79c6a923548aa99"
Jul 17 20:51:08.196681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:08.196652    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"
Jul 17 20:51:08.196870 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:08.196690    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp" event={"ID":"08c243ad-338e-4f7f-a75e-e34b5664585a","Type":"ContainerDied","Data":"408c48b502abba77de37c36935ce0b5d85002d9325543bf62feeb0e8d48bddd7"}
Jul 17 20:51:08.196870 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:08.196754    2572 scope.go:117] "RemoveContainer" containerID="8510e929bbba7baaaf9728f6d7c1b115d443b71823c40a12adedd97eb2c354fd"
Jul 17 20:51:08.198566 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:08.198533    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" event={"ID":"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f","Type":"ContainerStarted","Data":"7c668b64602cb33799ccffc22ecb7af35f094aedf72bf497b0d066aa3ed3240b"}
Jul 17 20:51:08.214176 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:08.214150    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"]
Jul 17 20:51:08.216752 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:08.216733    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-gkrbp"]
Jul 17 20:51:08.231961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:08.231923    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" podStartSLOduration=17.231910424 podStartE2EDuration="17.231910424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:51:08.231334224 +0000 UTC m=+3302.925310371" watchObservedRunningTime="2026-07-17 20:51:08.231910424 +0000 UTC m=+3302.925886568"
Jul 17 20:51:09.210205 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.210172    2572 generic.go:356] "Generic (PLEG): container finished" podID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerID="7c668b64602cb33799ccffc22ecb7af35f094aedf72bf497b0d066aa3ed3240b" exitCode=6
Jul 17 20:51:09.210661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.210242    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" event={"ID":"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f","Type":"ContainerDied","Data":"7c668b64602cb33799ccffc22ecb7af35f094aedf72bf497b0d066aa3ed3240b"}
Jul 17 20:51:09.210661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.210331    2572 scope.go:117] "RemoveContainer" containerID="e9bbb4b4ea2f7aa4d22ece8c4bea135aaeff0e33a2aee500f79c6a923548aa99"
Jul 17 20:51:09.210661 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.210646    2572 scope.go:117] "RemoveContainer" containerID="7c668b64602cb33799ccffc22ecb7af35f094aedf72bf497b0d066aa3ed3240b"
Jul 17 20:51:09.210869 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:09.210845    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-df4bb_odh-ai-gateway-infra(9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f"
Jul 17 20:51:09.241570 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.241539    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"]
Jul 17 20:51:09.853890 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.853856    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" path="/var/lib/kubelet/pods/08c243ad-338e-4f7f-a75e-e34b5664585a/volumes"
Jul 17 20:51:09.961469 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.961430    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"]
Jul 17 20:51:09.961753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.961742    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:09.961798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.961754    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:09.961798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.961778    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:09.961798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.961783    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:09.961887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.961836    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:09.961887 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.961846    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:09.966150 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.966133    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"
Jul 17 20:51:09.973866 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:09.973845    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"]
Jul 17 20:51:10.113266 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.113185    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2659cfb-9cc6-47ae-b88e-39c69d483204-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c\" (UID: \"b2659cfb-9cc6-47ae-b88e-39c69d483204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"
Jul 17 20:51:10.213663 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.213632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2659cfb-9cc6-47ae-b88e-39c69d483204-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c\" (UID: \"b2659cfb-9cc6-47ae-b88e-39c69d483204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"
Jul 17 20:51:10.214251 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.214225    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2659cfb-9cc6-47ae-b88e-39c69d483204-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c\" (UID: \"b2659cfb-9cc6-47ae-b88e-39c69d483204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"
Jul 17 20:51:10.276366 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.276323    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"
Jul 17 20:51:10.359619 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.359597    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"
Jul 17 20:51:10.415399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.415377    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"]
Jul 17 20:51:10.417308 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:51:10.417270    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2659cfb_9cc6_47ae_b88e_39c69d483204.slice/crio-8039aa232c09f40435a124453c2c4136a5ec413aefd264f7c4749bc4ac2f4e12 WatchSource:0}: Error finding container 8039aa232c09f40435a124453c2c4136a5ec413aefd264f7c4749bc4ac2f4e12: Status 404 returned error can't find the container with id 8039aa232c09f40435a124453c2c4136a5ec413aefd264f7c4749bc4ac2f4e12
Jul 17 20:51:10.516681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.516647    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f-maas-api-service-ca\") pod \"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f\" (UID: \"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f\") "
Jul 17 20:51:10.517030 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.517004    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" (UID: "9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:51:10.617443 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:10.617358    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:51:11.223688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.223656    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"
Jul 17 20:51:11.223688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.223660    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb" event={"ID":"9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f","Type":"ContainerDied","Data":"3843afaa4fa08713b982f9950b09c54ee3387216fb5877aa11d128b9de8eedfe"}
Jul 17 20:51:11.224250 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.223722    2572 scope.go:117] "RemoveContainer" containerID="7c668b64602cb33799ccffc22ecb7af35f094aedf72bf497b0d066aa3ed3240b"
Jul 17 20:51:11.225407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.225381    2572 generic.go:356] "Generic (PLEG): container finished" podID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerID="0520cc197e222f540616a2aae5015dcb5f67356087abd9b5372ad4a709119b84" exitCode=6
Jul 17 20:51:11.225532 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.225486    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" event={"ID":"b2659cfb-9cc6-47ae-b88e-39c69d483204","Type":"ContainerDied","Data":"0520cc197e222f540616a2aae5015dcb5f67356087abd9b5372ad4a709119b84"}
Jul 17 20:51:11.225532 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.225522    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" event={"ID":"b2659cfb-9cc6-47ae-b88e-39c69d483204","Type":"ContainerStarted","Data":"8039aa232c09f40435a124453c2c4136a5ec413aefd264f7c4749bc4ac2f4e12"}
Jul 17 20:51:11.225680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.225610    2572 scope.go:117] "RemoveContainer" containerID="0520cc197e222f540616a2aae5015dcb5f67356087abd9b5372ad4a709119b84"
Jul 17 20:51:11.253732 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.253711    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"]
Jul 17 20:51:11.259084 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.259063    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-df4bb"]
Jul 17 20:51:11.853100 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:11.853075    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" path="/var/lib/kubelet/pods/9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f/volumes"
Jul 17 20:51:12.232934 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:12.232901    2572 generic.go:356] "Generic (PLEG): container finished" podID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerID="3953305b8324fa0ba756cef1220845f397d7fe767b5689f0deb6baff36129809" exitCode=6
Jul 17 20:51:12.233407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:12.232973    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" event={"ID":"b2659cfb-9cc6-47ae-b88e-39c69d483204","Type":"ContainerDied","Data":"3953305b8324fa0ba756cef1220845f397d7fe767b5689f0deb6baff36129809"}
Jul 17 20:51:12.233407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:12.233023    2572 scope.go:117] "RemoveContainer" containerID="0520cc197e222f540616a2aae5015dcb5f67356087abd9b5372ad4a709119b84"
Jul 17 20:51:12.233407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:12.233272    2572 scope.go:117] "RemoveContainer" containerID="3953305b8324fa0ba756cef1220845f397d7fe767b5689f0deb6baff36129809"
Jul 17 20:51:12.233564 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:12.233480    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c_odh-ai-gateway-infra(b2659cfb-9cc6-47ae-b88e-39c69d483204)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204"
Jul 17 20:51:13.239280 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:13.239256    2572 scope.go:117] "RemoveContainer" containerID="3953305b8324fa0ba756cef1220845f397d7fe767b5689f0deb6baff36129809"
Jul 17 20:51:13.239662 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:13.239407    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c_odh-ai-gateway-infra(b2659cfb-9cc6-47ae-b88e-39c69d483204)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204"
Jul 17 20:51:13.848148 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:13.848119    2572 scope.go:117] "RemoveContainer" containerID="27552100bfb1655085c2ca9820eaea3dc82922d5b7ef2558a90a5704c4758a81"
Jul 17 20:51:14.245681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:14.245648    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" event={"ID":"6aace9f7-4168-4f44-8031-060b8e85aef8","Type":"ContainerStarted","Data":"969025a76d1c901ddc363a1d9c87050fb943479600ce5516891322dd93add1fc"}
Jul 17 20:51:15.251477 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:15.251446    2572 generic.go:356] "Generic (PLEG): container finished" podID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerID="969025a76d1c901ddc363a1d9c87050fb943479600ce5516891322dd93add1fc" exitCode=6
Jul 17 20:51:15.251905 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:15.251488    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" event={"ID":"6aace9f7-4168-4f44-8031-060b8e85aef8","Type":"ContainerDied","Data":"969025a76d1c901ddc363a1d9c87050fb943479600ce5516891322dd93add1fc"}
Jul 17 20:51:15.251905 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:15.251533    2572 scope.go:117] "RemoveContainer" containerID="27552100bfb1655085c2ca9820eaea3dc82922d5b7ef2558a90a5704c4758a81"
Jul 17 20:51:15.251905 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:15.251736    2572 scope.go:117] "RemoveContainer" containerID="969025a76d1c901ddc363a1d9c87050fb943479600ce5516891322dd93add1fc"
Jul 17 20:51:15.252031 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:15.251959    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w_odh-ai-gateway-infra(6aace9f7-4168-4f44-8031-060b8e85aef8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8"
Jul 17 20:51:15.275112 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:15.275084    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"]
Jul 17 20:51:16.384919 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:16.384899    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"
Jul 17 20:51:16.564727 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:16.564630    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aace9f7-4168-4f44-8031-060b8e85aef8-maas-api-service-ca\") pod \"6aace9f7-4168-4f44-8031-060b8e85aef8\" (UID: \"6aace9f7-4168-4f44-8031-060b8e85aef8\") "
Jul 17 20:51:16.565039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:16.565005    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aace9f7-4168-4f44-8031-060b8e85aef8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6aace9f7-4168-4f44-8031-060b8e85aef8" (UID: "6aace9f7-4168-4f44-8031-060b8e85aef8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:51:16.665828 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:16.665802    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aace9f7-4168-4f44-8031-060b8e85aef8-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:51:17.263563 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:17.263533    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"
Jul 17 20:51:17.263729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:17.263579    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w" event={"ID":"6aace9f7-4168-4f44-8031-060b8e85aef8","Type":"ContainerDied","Data":"9d3cfb684bcd61521fefb481660c245e981e89216aa37e456d7a1ae3245ba1df"}
Jul 17 20:51:17.263729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:17.263640    2572 scope.go:117] "RemoveContainer" containerID="969025a76d1c901ddc363a1d9c87050fb943479600ce5516891322dd93add1fc"
Jul 17 20:51:17.288098 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:17.288073    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"]
Jul 17 20:51:17.294934 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:17.294911    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-pjw5w"]
Jul 17 20:51:17.852788 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:17.852755    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" path="/var/lib/kubelet/pods/6aace9f7-4168-4f44-8031-060b8e85aef8/volumes"
Jul 17 20:51:26.783981 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:26.783941    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:51:28.848159 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:28.848080    2572 scope.go:117] "RemoveContainer" containerID="3953305b8324fa0ba756cef1220845f397d7fe767b5689f0deb6baff36129809"
Jul 17 20:51:29.324107 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:29.324068    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" event={"ID":"b2659cfb-9cc6-47ae-b88e-39c69d483204","Type":"ContainerStarted","Data":"fdc8093a4cc8dadb2941a940d4ab28e78887c07bf8d95d55cd5529a03be47c7e"}
Jul 17 20:51:29.341259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:29.341198    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" podStartSLOduration=20.341184091 podStartE2EDuration="20.341184091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:51:29.339783669 +0000 UTC m=+3324.033759816" watchObservedRunningTime="2026-07-17 20:51:29.341184091 +0000 UTC m=+3324.035160215"
Jul 17 20:51:30.331766 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:30.331734    2572 generic.go:356] "Generic (PLEG): container finished" podID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerID="fdc8093a4cc8dadb2941a940d4ab28e78887c07bf8d95d55cd5529a03be47c7e" exitCode=6
Jul 17 20:51:30.332290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:30.331773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" event={"ID":"b2659cfb-9cc6-47ae-b88e-39c69d483204","Type":"ContainerDied","Data":"fdc8093a4cc8dadb2941a940d4ab28e78887c07bf8d95d55cd5529a03be47c7e"}
Jul 17 20:51:30.332290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:30.331821    2572 scope.go:117] "RemoveContainer" containerID="3953305b8324fa0ba756cef1220845f397d7fe767b5689f0deb6baff36129809"
Jul 17 20:51:30.332290 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:30.332161    2572 scope.go:117] "RemoveContainer" containerID="fdc8093a4cc8dadb2941a940d4ab28e78887c07bf8d95d55cd5529a03be47c7e"
Jul 17 20:51:30.332454 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:30.332378    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c_odh-ai-gateway-infra(b2659cfb-9cc6-47ae-b88e-39c69d483204)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204"
Jul 17 20:51:30.352223 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:30.352198    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"]
Jul 17 20:51:31.475139 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:31.475116    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"
Jul 17 20:51:31.482367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:31.482342    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2659cfb-9cc6-47ae-b88e-39c69d483204-maas-api-service-ca\") pod \"b2659cfb-9cc6-47ae-b88e-39c69d483204\" (UID: \"b2659cfb-9cc6-47ae-b88e-39c69d483204\") "
Jul 17 20:51:31.482692 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:31.482673    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2659cfb-9cc6-47ae-b88e-39c69d483204-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2659cfb-9cc6-47ae-b88e-39c69d483204" (UID: "b2659cfb-9cc6-47ae-b88e-39c69d483204"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:51:31.583322 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:31.583288    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2659cfb-9cc6-47ae-b88e-39c69d483204-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:51:32.344118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:32.344080    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"
Jul 17 20:51:32.344118 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:32.344106    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c" event={"ID":"b2659cfb-9cc6-47ae-b88e-39c69d483204","Type":"ContainerDied","Data":"8039aa232c09f40435a124453c2c4136a5ec413aefd264f7c4749bc4ac2f4e12"}
Jul 17 20:51:32.344345 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:32.344167    2572 scope.go:117] "RemoveContainer" containerID="fdc8093a4cc8dadb2941a940d4ab28e78887c07bf8d95d55cd5529a03be47c7e"
Jul 17 20:51:32.359221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:32.359195    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"]
Jul 17 20:51:32.365184 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:32.365161    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qbk5c"]
Jul 17 20:51:33.852945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:33.852911    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" path="/var/lib/kubelet/pods/b2659cfb-9cc6-47ae-b88e-39c69d483204/volumes"
Jul 17 20:51:41.878662 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878629    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"]
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878933    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878944    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878957    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878962    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878973    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878978    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878987    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.878992    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879002    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879008    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879016    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879022    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879030    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879034    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879041    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879062    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879108    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879115    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879120    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879125    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879130    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879135    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879140    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879145    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879151    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2659cfb-9cc6-47ae-b88e-39c69d483204" containerName="revoke-keys"
Jul 17 20:51:41.879914 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.879157    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c243ad-338e-4f7f-a75e-e34b5664585a" containerName="revoke-keys"
Jul 17 20:51:41.882338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.882322    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"
Jul 17 20:51:41.884431 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.884414    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:51:41.888472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.888450    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"]
Jul 17 20:51:41.963147 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:41.963112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb72143-526b-40c9-941c-a1a3f55df7ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-czndk\" (UID: \"1cb72143-526b-40c9-941c-a1a3f55df7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"
Jul 17 20:51:42.064071 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:42.064008    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb72143-526b-40c9-941c-a1a3f55df7ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-czndk\" (UID: \"1cb72143-526b-40c9-941c-a1a3f55df7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"
Jul 17 20:51:42.064690 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:42.064663    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb72143-526b-40c9-941c-a1a3f55df7ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-czndk\" (UID: \"1cb72143-526b-40c9-941c-a1a3f55df7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"
Jul 17 20:51:42.194351 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:42.194255    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"
Jul 17 20:51:42.327797 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:42.327774    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"]
Jul 17 20:51:42.329856 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:51:42.329818    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb72143_526b_40c9_941c_a1a3f55df7ae.slice/crio-916f49e629756e379ed3546f721b321c24ed0eee3fbf59a6f3d351e2c37977d0 WatchSource:0}: Error finding container 916f49e629756e379ed3546f721b321c24ed0eee3fbf59a6f3d351e2c37977d0: Status 404 returned error can't find the container with id 916f49e629756e379ed3546f721b321c24ed0eee3fbf59a6f3d351e2c37977d0
Jul 17 20:51:42.396257 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:42.396224    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" event={"ID":"1cb72143-526b-40c9-941c-a1a3f55df7ae","Type":"ContainerStarted","Data":"916f49e629756e379ed3546f721b321c24ed0eee3fbf59a6f3d351e2c37977d0"}
Jul 17 20:51:43.402720 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:43.402682    2572 generic.go:356] "Generic (PLEG): container finished" podID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerID="c372ac22be96fb21dd063393039d579ae199795a5360c7845c581ec56dc2e224" exitCode=6
Jul 17 20:51:43.403153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:43.402771    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" event={"ID":"1cb72143-526b-40c9-941c-a1a3f55df7ae","Type":"ContainerDied","Data":"c372ac22be96fb21dd063393039d579ae199795a5360c7845c581ec56dc2e224"}
Jul 17 20:51:43.403153 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:43.402948    2572 scope.go:117] "RemoveContainer" containerID="c372ac22be96fb21dd063393039d579ae199795a5360c7845c581ec56dc2e224"
Jul 17 20:51:44.409830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:44.409743    2572 generic.go:356] "Generic (PLEG): container finished" podID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerID="c87a3f16f7bcd425897816b3660ab3e7d2d2bc0bba7c04daf747ed59339fb80b" exitCode=6
Jul 17 20:51:44.409830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:44.409819    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" event={"ID":"1cb72143-526b-40c9-941c-a1a3f55df7ae","Type":"ContainerDied","Data":"c87a3f16f7bcd425897816b3660ab3e7d2d2bc0bba7c04daf747ed59339fb80b"}
Jul 17 20:51:44.410329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:44.409873    2572 scope.go:117] "RemoveContainer" containerID="c372ac22be96fb21dd063393039d579ae199795a5360c7845c581ec56dc2e224"
Jul 17 20:51:44.410329 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:44.410118    2572 scope.go:117] "RemoveContainer" containerID="c87a3f16f7bcd425897816b3660ab3e7d2d2bc0bba7c04daf747ed59339fb80b"
Jul 17 20:51:44.410416 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:44.410348    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-czndk_odh-ai-gateway-infra(1cb72143-526b-40c9-941c-a1a3f55df7ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae"
Jul 17 20:51:45.416216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.416187    2572 scope.go:117] "RemoveContainer" containerID="c87a3f16f7bcd425897816b3660ab3e7d2d2bc0bba7c04daf747ed59339fb80b"
Jul 17 20:51:45.416576 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:45.416352    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-czndk_odh-ai-gateway-infra(1cb72143-526b-40c9-941c-a1a3f55df7ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae"
Jul 17 20:51:45.981765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.981717    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"]
Jul 17 20:51:45.982158 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.982138    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:45.982258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.982160    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9a17d6-8a0e-4ec9-a50f-c7a1344a802f" containerName="revoke-keys"
Jul 17 20:51:45.982258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.982188    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:45.982258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.982196    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aace9f7-4168-4f44-8031-060b8e85aef8" containerName="revoke-keys"
Jul 17 20:51:45.986448 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.986425    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"
Jul 17 20:51:45.996907 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:45.996879    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"]
Jul 17 20:51:46.099898 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:46.099861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2753b55-659a-414a-8f89-8bb842ef90e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr\" (UID: \"c2753b55-659a-414a-8f89-8bb842ef90e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"
Jul 17 20:51:46.201060 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:46.201022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2753b55-659a-414a-8f89-8bb842ef90e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr\" (UID: \"c2753b55-659a-414a-8f89-8bb842ef90e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"
Jul 17 20:51:46.201675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:46.201651    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2753b55-659a-414a-8f89-8bb842ef90e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr\" (UID: \"c2753b55-659a-414a-8f89-8bb842ef90e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"
Jul 17 20:51:46.296974 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:46.296888    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"
Jul 17 20:51:46.432677 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:46.432654    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"]
Jul 17 20:51:46.434883 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:51:46.434859    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2753b55_659a_414a_8f89_8bb842ef90e4.slice/crio-694e115d3b6597333ee4cd014f9c557eb5a944cde1a259744dbecb250c48d73d WatchSource:0}: Error finding container 694e115d3b6597333ee4cd014f9c557eb5a944cde1a259744dbecb250c48d73d: Status 404 returned error can't find the container with id 694e115d3b6597333ee4cd014f9c557eb5a944cde1a259744dbecb250c48d73d
Jul 17 20:51:47.428169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:47.428075    2572 generic.go:356] "Generic (PLEG): container finished" podID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerID="269fafbe8a1a987a605e7f882e10dcc3aa6f1f2d7642be52fab7a67da2511d96" exitCode=6
Jul 17 20:51:47.428169 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:47.428134    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" event={"ID":"c2753b55-659a-414a-8f89-8bb842ef90e4","Type":"ContainerDied","Data":"269fafbe8a1a987a605e7f882e10dcc3aa6f1f2d7642be52fab7a67da2511d96"}
Jul 17 20:51:47.428680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:47.428196    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" event={"ID":"c2753b55-659a-414a-8f89-8bb842ef90e4","Type":"ContainerStarted","Data":"694e115d3b6597333ee4cd014f9c557eb5a944cde1a259744dbecb250c48d73d"}
Jul 17 20:51:47.428680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:47.428438    2572 scope.go:117] "RemoveContainer" containerID="269fafbe8a1a987a605e7f882e10dcc3aa6f1f2d7642be52fab7a67da2511d96"
Jul 17 20:51:48.201063 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.201001    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"]
Jul 17 20:51:48.204434 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.204415    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"
Jul 17 20:51:48.210826 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.210805    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"]
Jul 17 20:51:48.320611 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.320519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc177a82-58ed-4b62-9364-be9d0be4c924-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-bltch\" (UID: \"fc177a82-58ed-4b62-9364-be9d0be4c924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"
Jul 17 20:51:48.421302 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.421264    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc177a82-58ed-4b62-9364-be9d0be4c924-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-bltch\" (UID: \"fc177a82-58ed-4b62-9364-be9d0be4c924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"
Jul 17 20:51:48.421890 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.421869    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc177a82-58ed-4b62-9364-be9d0be4c924-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-bltch\" (UID: \"fc177a82-58ed-4b62-9364-be9d0be4c924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"
Jul 17 20:51:48.436349 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.436319    2572 generic.go:356] "Generic (PLEG): container finished" podID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerID="7dfd0147e66e2488f96082dfe0799799a66b439ef0c1ea00f4066fc95b861c5b" exitCode=6
Jul 17 20:51:48.436527 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.436412    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" event={"ID":"c2753b55-659a-414a-8f89-8bb842ef90e4","Type":"ContainerDied","Data":"7dfd0147e66e2488f96082dfe0799799a66b439ef0c1ea00f4066fc95b861c5b"}
Jul 17 20:51:48.436527 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.436476    2572 scope.go:117] "RemoveContainer" containerID="269fafbe8a1a987a605e7f882e10dcc3aa6f1f2d7642be52fab7a67da2511d96"
Jul 17 20:51:48.436656 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.436586    2572 scope.go:117] "RemoveContainer" containerID="7dfd0147e66e2488f96082dfe0799799a66b439ef0c1ea00f4066fc95b861c5b"
Jul 17 20:51:48.436807 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:48.436785    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr_odh-ai-gateway-infra(c2753b55-659a-414a-8f89-8bb842ef90e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4"
Jul 17 20:51:48.515679 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.515647    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"
Jul 17 20:51:48.673664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:48.673642    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"]
Jul 17 20:51:48.675957 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:51:48.675928    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc177a82_58ed_4b62_9364_be9d0be4c924.slice/crio-616dc39d7d83b4cb3faf31380df5c96847b191909770520bcd81c7af10e60e9c WatchSource:0}: Error finding container 616dc39d7d83b4cb3faf31380df5c96847b191909770520bcd81c7af10e60e9c: Status 404 returned error can't find the container with id 616dc39d7d83b4cb3faf31380df5c96847b191909770520bcd81c7af10e60e9c
Jul 17 20:51:49.443178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:49.443147    2572 scope.go:117] "RemoveContainer" containerID="7dfd0147e66e2488f96082dfe0799799a66b439ef0c1ea00f4066fc95b861c5b"
Jul 17 20:51:49.443645 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:49.443343    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr_odh-ai-gateway-infra(c2753b55-659a-414a-8f89-8bb842ef90e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4"
Jul 17 20:51:49.444438 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:49.444417    2572 generic.go:356] "Generic (PLEG): container finished" podID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerID="4cae27929f7fef36ebef2987a366db34d5578b69e2fc9587f6552baadd560a84" exitCode=6
Jul 17 20:51:49.444493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:49.444465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" event={"ID":"fc177a82-58ed-4b62-9364-be9d0be4c924","Type":"ContainerDied","Data":"4cae27929f7fef36ebef2987a366db34d5578b69e2fc9587f6552baadd560a84"}
Jul 17 20:51:49.444527 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:49.444505    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" event={"ID":"fc177a82-58ed-4b62-9364-be9d0be4c924","Type":"ContainerStarted","Data":"616dc39d7d83b4cb3faf31380df5c96847b191909770520bcd81c7af10e60e9c"}
Jul 17 20:51:49.444699 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:49.444687    2572 scope.go:117] "RemoveContainer" containerID="4cae27929f7fef36ebef2987a366db34d5578b69e2fc9587f6552baadd560a84"
Jul 17 20:51:49.593057 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:49.593002    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:51:50.452510 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:50.452421    2572 generic.go:356] "Generic (PLEG): container finished" podID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerID="54f914b0f8f4aa82f41b717e8ea8186766df2b89373464871f1c5656632a7c01" exitCode=6
Jul 17 20:51:50.452940 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:50.452511    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" event={"ID":"fc177a82-58ed-4b62-9364-be9d0be4c924","Type":"ContainerDied","Data":"54f914b0f8f4aa82f41b717e8ea8186766df2b89373464871f1c5656632a7c01"}
Jul 17 20:51:50.452940 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:50.452590    2572 scope.go:117] "RemoveContainer" containerID="4cae27929f7fef36ebef2987a366db34d5578b69e2fc9587f6552baadd560a84"
Jul 17 20:51:50.452940 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:50.452809    2572 scope.go:117] "RemoveContainer" containerID="54f914b0f8f4aa82f41b717e8ea8186766df2b89373464871f1c5656632a7c01"
Jul 17 20:51:50.453079 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:50.452994    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-bltch_odh-ai-gateway-infra(fc177a82-58ed-4b62-9364-be9d0be4c924)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924"
Jul 17 20:51:51.459662 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:51.459635    2572 scope.go:117] "RemoveContainer" containerID="54f914b0f8f4aa82f41b717e8ea8186766df2b89373464871f1c5656632a7c01"
Jul 17 20:51:51.460036 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:51.459819    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-bltch_odh-ai-gateway-infra(fc177a82-58ed-4b62-9364-be9d0be4c924)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924"
Jul 17 20:51:58.848338 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:58.848305    2572 scope.go:117] "RemoveContainer" containerID="c87a3f16f7bcd425897816b3660ab3e7d2d2bc0bba7c04daf747ed59339fb80b"
Jul 17 20:51:59.499841 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:59.499814    2572 generic.go:356] "Generic (PLEG): container finished" podID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerID="5b7520b72eb4e355bc889950a781e3a42529d5abb63ed4f8f52f236dfca77038" exitCode=6
Jul 17 20:51:59.500015 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:59.499857    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" event={"ID":"1cb72143-526b-40c9-941c-a1a3f55df7ae","Type":"ContainerDied","Data":"5b7520b72eb4e355bc889950a781e3a42529d5abb63ed4f8f52f236dfca77038"}
Jul 17 20:51:59.500015 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:59.499902    2572 scope.go:117] "RemoveContainer" containerID="c87a3f16f7bcd425897816b3660ab3e7d2d2bc0bba7c04daf747ed59339fb80b"
Jul 17 20:51:59.500270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:51:59.500244    2572 scope.go:117] "RemoveContainer" containerID="5b7520b72eb4e355bc889950a781e3a42529d5abb63ed4f8f52f236dfca77038"
Jul 17 20:51:59.500467 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:51:59.500447    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-czndk_odh-ai-gateway-infra(1cb72143-526b-40c9-941c-a1a3f55df7ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae"
Jul 17 20:52:00.527347 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:00.527309    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"]
Jul 17 20:52:00.665916 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:00.665894    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"
Jul 17 20:52:00.734098 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:00.734070    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb72143-526b-40c9-941c-a1a3f55df7ae-maas-api-service-ca\") pod \"1cb72143-526b-40c9-941c-a1a3f55df7ae\" (UID: \"1cb72143-526b-40c9-941c-a1a3f55df7ae\") "
Jul 17 20:52:00.734475 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:00.734444    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb72143-526b-40c9-941c-a1a3f55df7ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1cb72143-526b-40c9-941c-a1a3f55df7ae" (UID: "1cb72143-526b-40c9-941c-a1a3f55df7ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:00.834711 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:00.834633    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb72143-526b-40c9-941c-a1a3f55df7ae-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:00.847886 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:00.847867    2572 scope.go:117] "RemoveContainer" containerID="7dfd0147e66e2488f96082dfe0799799a66b439ef0c1ea00f4066fc95b861c5b"
Jul 17 20:52:01.512274 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.512243    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"
Jul 17 20:52:01.512274 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.512261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk" event={"ID":"1cb72143-526b-40c9-941c-a1a3f55df7ae","Type":"ContainerDied","Data":"916f49e629756e379ed3546f721b321c24ed0eee3fbf59a6f3d351e2c37977d0"}
Jul 17 20:52:01.512505 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.512324    2572 scope.go:117] "RemoveContainer" containerID="5b7520b72eb4e355bc889950a781e3a42529d5abb63ed4f8f52f236dfca77038"
Jul 17 20:52:01.514368 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.514336    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" event={"ID":"c2753b55-659a-414a-8f89-8bb842ef90e4","Type":"ContainerStarted","Data":"7f43f57cde71252babffeb7a5fcaba87ea683249772c51723c4a3b32bef1b880"}
Jul 17 20:52:01.536931 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.536887    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" podStartSLOduration=16.536872006 podStartE2EDuration="16.536872006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:52:01.535157339 +0000 UTC m=+3356.229133486" watchObservedRunningTime="2026-07-17 20:52:01.536872006 +0000 UTC m=+3356.230848194"
Jul 17 20:52:01.548680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.548657    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"]
Jul 17 20:52:01.551999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.551976    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-czndk"]
Jul 17 20:52:01.852954 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:01.852862    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" path="/var/lib/kubelet/pods/1cb72143-526b-40c9-941c-a1a3f55df7ae/volumes"
Jul 17 20:52:02.521088 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:02.521058    2572 generic.go:356] "Generic (PLEG): container finished" podID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerID="7f43f57cde71252babffeb7a5fcaba87ea683249772c51723c4a3b32bef1b880" exitCode=6
Jul 17 20:52:02.521088 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:02.521077    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" event={"ID":"c2753b55-659a-414a-8f89-8bb842ef90e4","Type":"ContainerDied","Data":"7f43f57cde71252babffeb7a5fcaba87ea683249772c51723c4a3b32bef1b880"}
Jul 17 20:52:02.521348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:02.521136    2572 scope.go:117] "RemoveContainer" containerID="7dfd0147e66e2488f96082dfe0799799a66b439ef0c1ea00f4066fc95b861c5b"
Jul 17 20:52:02.521444 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:02.521425    2572 scope.go:117] "RemoveContainer" containerID="7f43f57cde71252babffeb7a5fcaba87ea683249772c51723c4a3b32bef1b880"
Jul 17 20:52:02.521685 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:02.521657    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr_odh-ai-gateway-infra(c2753b55-659a-414a-8f89-8bb842ef90e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4"
Jul 17 20:52:02.544910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:02.544885    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"]
Jul 17 20:52:02.848102 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:02.848005    2572 scope.go:117] "RemoveContainer" containerID="54f914b0f8f4aa82f41b717e8ea8186766df2b89373464871f1c5656632a7c01"
Jul 17 20:52:03.531886 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:03.531853    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" event={"ID":"fc177a82-58ed-4b62-9364-be9d0be4c924","Type":"ContainerStarted","Data":"4d1b5cbfe2fe7e93ea352d35682a2c00a9a768b5a0291935f6f47af12b9a8ef7"}
Jul 17 20:52:03.561946 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:03.561899    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" podStartSLOduration=15.561882069 podStartE2EDuration="15.561882069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:52:03.560791025 +0000 UTC m=+3358.254767165" watchObservedRunningTime="2026-07-17 20:52:03.561882069 +0000 UTC m=+3358.255858214"
Jul 17 20:52:03.661343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:03.661322    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"
Jul 17 20:52:03.759005 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:03.758975    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2753b55-659a-414a-8f89-8bb842ef90e4-maas-api-service-ca\") pod \"c2753b55-659a-414a-8f89-8bb842ef90e4\" (UID: \"c2753b55-659a-414a-8f89-8bb842ef90e4\") "
Jul 17 20:52:03.759347 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:03.759324    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2753b55-659a-414a-8f89-8bb842ef90e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2753b55-659a-414a-8f89-8bb842ef90e4" (UID: "c2753b55-659a-414a-8f89-8bb842ef90e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:03.859471 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:03.859404    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2753b55-659a-414a-8f89-8bb842ef90e4-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:04.538027 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.537997    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"
Jul 17 20:52:04.538252 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.538033    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr" event={"ID":"c2753b55-659a-414a-8f89-8bb842ef90e4","Type":"ContainerDied","Data":"694e115d3b6597333ee4cd014f9c557eb5a944cde1a259744dbecb250c48d73d"}
Jul 17 20:52:04.538252 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.538138    2572 scope.go:117] "RemoveContainer" containerID="7f43f57cde71252babffeb7a5fcaba87ea683249772c51723c4a3b32bef1b880"
Jul 17 20:52:04.539834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.539807    2572 generic.go:356] "Generic (PLEG): container finished" podID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerID="4d1b5cbfe2fe7e93ea352d35682a2c00a9a768b5a0291935f6f47af12b9a8ef7" exitCode=6
Jul 17 20:52:04.539952 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.539887    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" event={"ID":"fc177a82-58ed-4b62-9364-be9d0be4c924","Type":"ContainerDied","Data":"4d1b5cbfe2fe7e93ea352d35682a2c00a9a768b5a0291935f6f47af12b9a8ef7"}
Jul 17 20:52:04.540302 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.540252    2572 scope.go:117] "RemoveContainer" containerID="4d1b5cbfe2fe7e93ea352d35682a2c00a9a768b5a0291935f6f47af12b9a8ef7"
Jul 17 20:52:04.540503 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:04.540480    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-bltch_odh-ai-gateway-infra(fc177a82-58ed-4b62-9364-be9d0be4c924)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924"
Jul 17 20:52:04.546451 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.546434    2572 scope.go:117] "RemoveContainer" containerID="54f914b0f8f4aa82f41b717e8ea8186766df2b89373464871f1c5656632a7c01"
Jul 17 20:52:04.553407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.553388    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"]
Jul 17 20:52:04.559275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.559254    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-hz2pr"]
Jul 17 20:52:04.569691 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:04.569670    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"]
Jul 17 20:52:05.180633 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:05.180599    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:52:05.674789 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:05.674768    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"
Jul 17 20:52:05.776239 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:05.776209    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc177a82-58ed-4b62-9364-be9d0be4c924-maas-api-service-ca\") pod \"fc177a82-58ed-4b62-9364-be9d0be4c924\" (UID: \"fc177a82-58ed-4b62-9364-be9d0be4c924\") "
Jul 17 20:52:05.776516 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:05.776494    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc177a82-58ed-4b62-9364-be9d0be4c924-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc177a82-58ed-4b62-9364-be9d0be4c924" (UID: "fc177a82-58ed-4b62-9364-be9d0be4c924"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:05.853630 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:05.853558    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" path="/var/lib/kubelet/pods/c2753b55-659a-414a-8f89-8bb842ef90e4/volumes"
Jul 17 20:52:05.877671 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:05.877642    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc177a82-58ed-4b62-9364-be9d0be4c924-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:06.553430 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:06.553394    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"
Jul 17 20:52:06.553633 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:06.553393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch" event={"ID":"fc177a82-58ed-4b62-9364-be9d0be4c924","Type":"ContainerDied","Data":"616dc39d7d83b4cb3faf31380df5c96847b191909770520bcd81c7af10e60e9c"}
Jul 17 20:52:06.553633 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:06.553550    2572 scope.go:117] "RemoveContainer" containerID="4d1b5cbfe2fe7e93ea352d35682a2c00a9a768b5a0291935f6f47af12b9a8ef7"
Jul 17 20:52:06.580609 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:06.580576    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"]
Jul 17 20:52:06.586654 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:06.586632    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-bltch"]
Jul 17 20:52:07.853543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:07.853506    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" path="/var/lib/kubelet/pods/fc177a82-58ed-4b62-9364-be9d0be4c924/volumes"
Jul 17 20:52:10.027550 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027515    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"]
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027877    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027893    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027901    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027908    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027937    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027943    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027953    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027963    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027972    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027978    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027984    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027989    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.027994    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.028000    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.028074    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:10.028073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.028085    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.028648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.028092    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:10.028648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.028098    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:10.028648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.028103    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:10.028648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.028110    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:10.032521 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.032499    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"
Jul 17 20:52:10.035622 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.035605    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:52:10.042188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.042164    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"]
Jul 17 20:52:10.115673 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.115635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a222c5-fa63-4b5b-a4b3-0e9f068978a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg\" (UID: \"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"
Jul 17 20:52:10.216216 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.216177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a222c5-fa63-4b5b-a4b3-0e9f068978a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg\" (UID: \"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"
Jul 17 20:52:10.216775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.216757    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a222c5-fa63-4b5b-a4b3-0e9f068978a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg\" (UID: \"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"
Jul 17 20:52:10.343872 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.343785    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"
Jul 17 20:52:10.475768 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.475744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"]
Jul 17 20:52:10.477587 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:10.477556    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a222c5_fa63_4b5b_a4b3_0e9f068978a1.slice/crio-9c7a5f9e4d745b99ad783d340a9850237362cba698d463232803312accd675c5 WatchSource:0}: Error finding container 9c7a5f9e4d745b99ad783d340a9850237362cba698d463232803312accd675c5: Status 404 returned error can't find the container with id 9c7a5f9e4d745b99ad783d340a9850237362cba698d463232803312accd675c5
Jul 17 20:52:10.575252 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:10.575216    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" event={"ID":"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1","Type":"ContainerStarted","Data":"9c7a5f9e4d745b99ad783d340a9850237362cba698d463232803312accd675c5"}
Jul 17 20:52:11.581403 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:11.581366    2572 generic.go:356] "Generic (PLEG): container finished" podID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerID="708bb0a01357b20ef94adf9d63977589f703585bd745b4209985acad61aa196a" exitCode=6
Jul 17 20:52:11.581790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:11.581443    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" event={"ID":"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1","Type":"ContainerDied","Data":"708bb0a01357b20ef94adf9d63977589f703585bd745b4209985acad61aa196a"}
Jul 17 20:52:11.581790 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:11.581667    2572 scope.go:117] "RemoveContainer" containerID="708bb0a01357b20ef94adf9d63977589f703585bd745b4209985acad61aa196a"
Jul 17 20:52:12.587957 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:12.587923    2572 generic.go:356] "Generic (PLEG): container finished" podID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerID="054710a065be2702792fcfa22a84e7eb2456f0af2d3201fcf5fd45751862c1ad" exitCode=6
Jul 17 20:52:12.588423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:12.588005    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" event={"ID":"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1","Type":"ContainerDied","Data":"054710a065be2702792fcfa22a84e7eb2456f0af2d3201fcf5fd45751862c1ad"}
Jul 17 20:52:12.588423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:12.588076    2572 scope.go:117] "RemoveContainer" containerID="708bb0a01357b20ef94adf9d63977589f703585bd745b4209985acad61aa196a"
Jul 17 20:52:12.588423 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:12.588194    2572 scope.go:117] "RemoveContainer" containerID="054710a065be2702792fcfa22a84e7eb2456f0af2d3201fcf5fd45751862c1ad"
Jul 17 20:52:12.588423 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:12.588404    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg_odh-ai-gateway-infra(a0a222c5-fa63-4b5b-a4b3-0e9f068978a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1"
Jul 17 20:52:13.593909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:13.593880    2572 scope.go:117] "RemoveContainer" containerID="054710a065be2702792fcfa22a84e7eb2456f0af2d3201fcf5fd45751862c1ad"
Jul 17 20:52:13.594318 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:13.594085    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg_odh-ai-gateway-infra(a0a222c5-fa63-4b5b-a4b3-0e9f068978a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1"
Jul 17 20:52:14.793246 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793206    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-140296589114288"]
Jul 17 20:52:14.793712 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793695    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:14.793784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793716    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:14.793784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793733    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:14.793784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793742    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:14.793945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793831    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc177a82-58ed-4b62-9364-be9d0be4c924" containerName="revoke-keys"
Jul 17 20:52:14.793945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793841    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cb72143-526b-40c9-941c-a1a3f55df7ae" containerName="revoke-keys"
Jul 17 20:52:14.793945 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.793854    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2753b55-659a-414a-8f89-8bb842ef90e4" containerName="revoke-keys"
Jul 17 20:52:14.798156 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.798134    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589114288"
Jul 17 20:52:14.800992 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.800969    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f99dr\""
Jul 17 20:52:14.802109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.802084    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589114288"]
Jul 17 20:52:14.957620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:14.957582    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498vj\" (UniqueName: \"kubernetes.io/projected/2445d804-9326-4fb2-8353-8ef362ea1e35-kube-api-access-498vj\") pod \"test-curl-7635-140296589114288\" (UID: \"2445d804-9326-4fb2-8353-8ef362ea1e35\") " pod="opendatahub/test-curl-7635-140296589114288"
Jul 17 20:52:15.059009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:15.058918    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498vj\" (UniqueName: \"kubernetes.io/projected/2445d804-9326-4fb2-8353-8ef362ea1e35-kube-api-access-498vj\") pod \"test-curl-7635-140296589114288\" (UID: \"2445d804-9326-4fb2-8353-8ef362ea1e35\") " pod="opendatahub/test-curl-7635-140296589114288"
Jul 17 20:52:15.067269 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:15.067247    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-498vj\" (UniqueName: \"kubernetes.io/projected/2445d804-9326-4fb2-8353-8ef362ea1e35-kube-api-access-498vj\") pod \"test-curl-7635-140296589114288\" (UID: \"2445d804-9326-4fb2-8353-8ef362ea1e35\") " pod="opendatahub/test-curl-7635-140296589114288"
Jul 17 20:52:15.107780 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:15.107749    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589114288"
Jul 17 20:52:15.245961 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:15.245936    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589114288"]
Jul 17 20:52:15.248490 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:15.248464    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2445d804_9326_4fb2_8353_8ef362ea1e35.slice/crio-f47f37d0c485e99b2fe1a97f7c5df2f23f7073d8b7ccbdd25276994fea873758 WatchSource:0}: Error finding container f47f37d0c485e99b2fe1a97f7c5df2f23f7073d8b7ccbdd25276994fea873758: Status 404 returned error can't find the container with id f47f37d0c485e99b2fe1a97f7c5df2f23f7073d8b7ccbdd25276994fea873758
Jul 17 20:52:15.605402 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:15.605365    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589114288" event={"ID":"2445d804-9326-4fb2-8353-8ef362ea1e35","Type":"ContainerStarted","Data":"f47f37d0c485e99b2fe1a97f7c5df2f23f7073d8b7ccbdd25276994fea873758"}
Jul 17 20:52:16.611518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:16.611485    2572 generic.go:356] "Generic (PLEG): container finished" podID="2445d804-9326-4fb2-8353-8ef362ea1e35" containerID="0395609425fe4321730d3f0e7a8ab6e968d5aef0bf54a9ed5eccc6968d5a3d14" exitCode=0
Jul 17 20:52:16.611518 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:16.611514    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589114288" event={"ID":"2445d804-9326-4fb2-8353-8ef362ea1e35","Type":"ContainerDied","Data":"0395609425fe4321730d3f0e7a8ab6e968d5aef0bf54a9ed5eccc6968d5a3d14"}
Jul 17 20:52:17.751275 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.751246    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589114288"
Jul 17 20:52:17.789165 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.789133    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-140296589114288_2445d804-9326-4fb2-8353-8ef362ea1e35/test-curl-7635-140296589114288/0.log"
Jul 17 20:52:17.811553 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.811501    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-140296589114288"]
Jul 17 20:52:17.813303 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.813278    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-140296589114288"]
Jul 17 20:52:17.881623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.881540    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-498vj\" (UniqueName: \"kubernetes.io/projected/2445d804-9326-4fb2-8353-8ef362ea1e35-kube-api-access-498vj\") pod \"2445d804-9326-4fb2-8353-8ef362ea1e35\" (UID: \"2445d804-9326-4fb2-8353-8ef362ea1e35\") "
Jul 17 20:52:17.883775 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.883739    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2445d804-9326-4fb2-8353-8ef362ea1e35-kube-api-access-498vj" (OuterVolumeSpecName: "kube-api-access-498vj") pod "2445d804-9326-4fb2-8353-8ef362ea1e35" (UID: "2445d804-9326-4fb2-8353-8ef362ea1e35"). InnerVolumeSpecName "kube-api-access-498vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:52:17.928843 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.928783    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-140296588943664"]
Jul 17 20:52:17.929230 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.929213    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2445d804-9326-4fb2-8353-8ef362ea1e35" containerName="test-curl-7635-140296589114288"
Jul 17 20:52:17.929278 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.929232    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2445d804-9326-4fb2-8353-8ef362ea1e35" containerName="test-curl-7635-140296589114288"
Jul 17 20:52:17.929325 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.929301    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2445d804-9326-4fb2-8353-8ef362ea1e35" containerName="test-curl-7635-140296589114288"
Jul 17 20:52:17.932320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.932303    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:17.940100 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.940074    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296588943664"]
Jul 17 20:52:17.988270 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:17.983579    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-498vj\" (UniqueName: \"kubernetes.io/projected/2445d804-9326-4fb2-8353-8ef362ea1e35-kube-api-access-498vj\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:18.084337 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.084295    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pndp\" (UniqueName: \"kubernetes.io/projected/ebd8f313-ce6d-4fbc-81ad-6ef113127d51-kube-api-access-9pndp\") pod \"test-curl-7635-140296588943664\" (UID: \"ebd8f313-ce6d-4fbc-81ad-6ef113127d51\") " pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:18.184941 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.184853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pndp\" (UniqueName: \"kubernetes.io/projected/ebd8f313-ce6d-4fbc-81ad-6ef113127d51-kube-api-access-9pndp\") pod \"test-curl-7635-140296588943664\" (UID: \"ebd8f313-ce6d-4fbc-81ad-6ef113127d51\") " pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:18.195343 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.195314    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pndp\" (UniqueName: \"kubernetes.io/projected/ebd8f313-ce6d-4fbc-81ad-6ef113127d51-kube-api-access-9pndp\") pod \"test-curl-7635-140296588943664\" (UID: \"ebd8f313-ce6d-4fbc-81ad-6ef113127d51\") " pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:18.242070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.242020    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:18.374187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.374162    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296588943664"]
Jul 17 20:52:18.375784 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:18.375757    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebd8f313_ce6d_4fbc_81ad_6ef113127d51.slice/crio-abeb75b947dc651d666c02799b880655951b81e0d814456b533fb21fca441ac1 WatchSource:0}: Error finding container abeb75b947dc651d666c02799b880655951b81e0d814456b533fb21fca441ac1: Status 404 returned error can't find the container with id abeb75b947dc651d666c02799b880655951b81e0d814456b533fb21fca441ac1
Jul 17 20:52:18.621635 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.621599    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589114288"
Jul 17 20:52:18.621635 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.621631    2572 scope.go:117] "RemoveContainer" containerID="0395609425fe4321730d3f0e7a8ab6e968d5aef0bf54a9ed5eccc6968d5a3d14"
Jul 17 20:52:18.622925 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:18.622753    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296588943664" event={"ID":"ebd8f313-ce6d-4fbc-81ad-6ef113127d51","Type":"ContainerStarted","Data":"abeb75b947dc651d666c02799b880655951b81e0d814456b533fb21fca441ac1"}
Jul 17 20:52:19.630374 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:19.630341    2572 generic.go:356] "Generic (PLEG): container finished" podID="ebd8f313-ce6d-4fbc-81ad-6ef113127d51" containerID="72c2ac088182fefc48b10f3708d913dd61e9321f4a2db08a6dc19615d6f88707" exitCode=0
Jul 17 20:52:19.630863 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:19.630387    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296588943664" event={"ID":"ebd8f313-ce6d-4fbc-81ad-6ef113127d51","Type":"ContainerDied","Data":"72c2ac088182fefc48b10f3708d913dd61e9321f4a2db08a6dc19615d6f88707"}
Jul 17 20:52:19.777129 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:19.777095    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:52:19.853620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:19.853587    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2445d804-9326-4fb2-8353-8ef362ea1e35" path="/var/lib/kubelet/pods/2445d804-9326-4fb2-8353-8ef362ea1e35/volumes"
Jul 17 20:52:20.762862 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.762841    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:20.782878 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.782850    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-140296588943664_ebd8f313-ce6d-4fbc-81ad-6ef113127d51/test-curl-7635-140296588943664/0.log"
Jul 17 20:52:20.804929 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.804898    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-140296588943664"]
Jul 17 20:52:20.808397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.808377    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-140296588943664"]
Jul 17 20:52:20.914860 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.914776    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pndp\" (UniqueName: \"kubernetes.io/projected/ebd8f313-ce6d-4fbc-81ad-6ef113127d51-kube-api-access-9pndp\") pod \"ebd8f313-ce6d-4fbc-81ad-6ef113127d51\" (UID: \"ebd8f313-ce6d-4fbc-81ad-6ef113127d51\") "
Jul 17 20:52:20.916839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.916804    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd8f313-ce6d-4fbc-81ad-6ef113127d51-kube-api-access-9pndp" (OuterVolumeSpecName: "kube-api-access-9pndp") pod "ebd8f313-ce6d-4fbc-81ad-6ef113127d51" (UID: "ebd8f313-ce6d-4fbc-81ad-6ef113127d51"). InnerVolumeSpecName "kube-api-access-9pndp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:52:20.964397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.964361    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:20.964731 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.964717    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd8f313-ce6d-4fbc-81ad-6ef113127d51" containerName="test-curl-7635-140296588943664"
Jul 17 20:52:20.964792 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.964732    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd8f313-ce6d-4fbc-81ad-6ef113127d51" containerName="test-curl-7635-140296588943664"
Jul 17 20:52:20.964834 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.964801    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd8f313-ce6d-4fbc-81ad-6ef113127d51" containerName="test-curl-7635-140296588943664"
Jul 17 20:52:20.967705 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.967688    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:20.971949 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:20.971926    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:21.016069 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.016015    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pndp\" (UniqueName: \"kubernetes.io/projected/ebd8f313-ce6d-4fbc-81ad-6ef113127d51-kube-api-access-9pndp\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:21.117106 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.117066    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84j4\" (UniqueName: \"kubernetes.io/projected/20cf3342-2cba-4180-93a3-fad3d205b2d5-kube-api-access-l84j4\") pod \"test-curl-7635-140296589198656\" (UID: \"20cf3342-2cba-4180-93a3-fad3d205b2d5\") " pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:21.218348 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.218318    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84j4\" (UniqueName: \"kubernetes.io/projected/20cf3342-2cba-4180-93a3-fad3d205b2d5-kube-api-access-l84j4\") pod \"test-curl-7635-140296589198656\" (UID: \"20cf3342-2cba-4180-93a3-fad3d205b2d5\") " pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:21.227856 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.227825    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84j4\" (UniqueName: \"kubernetes.io/projected/20cf3342-2cba-4180-93a3-fad3d205b2d5-kube-api-access-l84j4\") pod \"test-curl-7635-140296589198656\" (UID: \"20cf3342-2cba-4180-93a3-fad3d205b2d5\") " pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:21.278618 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.278588    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:21.408645 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.408617    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:21.410513 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:21.410484    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20cf3342_2cba_4180_93a3_fad3d205b2d5.slice/crio-9305559d8e6711aed92beb39d121668cd3535b656f078cb08cbf288f4f8cabc5 WatchSource:0}: Error finding container 9305559d8e6711aed92beb39d121668cd3535b656f078cb08cbf288f4f8cabc5: Status 404 returned error can't find the container with id 9305559d8e6711aed92beb39d121668cd3535b656f078cb08cbf288f4f8cabc5
Jul 17 20:52:21.641964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.641879    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:21.641964 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.641889    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abeb75b947dc651d666c02799b880655951b81e0d814456b533fb21fca441ac1"
Jul 17 20:52:21.643103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.643073    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589198656" event={"ID":"20cf3342-2cba-4180-93a3-fad3d205b2d5","Type":"ContainerStarted","Data":"9305559d8e6711aed92beb39d121668cd3535b656f078cb08cbf288f4f8cabc5"}
Jul 17 20:52:21.652243 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:21.652214    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-140296588943664\" is forbidden: User \"system:node:ip-10-0-143-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-185.ec2.internal' and this object" podUID="ebd8f313-ce6d-4fbc-81ad-6ef113127d51" pod="opendatahub/test-curl-7635-140296588943664"
Jul 17 20:52:21.853565 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:21.853535    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd8f313-ce6d-4fbc-81ad-6ef113127d51" path="/var/lib/kubelet/pods/ebd8f313-ce6d-4fbc-81ad-6ef113127d51/volumes"
Jul 17 20:52:22.650070 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:22.650011    2572 generic.go:356] "Generic (PLEG): container finished" podID="20cf3342-2cba-4180-93a3-fad3d205b2d5" containerID="9c021e3215f918332c9a914f5915b096404d95bea7b5204c5ac606d982aef03c" exitCode=0
Jul 17 20:52:22.650245 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:22.650090    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589198656" event={"ID":"20cf3342-2cba-4180-93a3-fad3d205b2d5","Type":"ContainerDied","Data":"9c021e3215f918332c9a914f5915b096404d95bea7b5204c5ac606d982aef03c"}
Jul 17 20:52:23.784951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.784926    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:23.814221 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.814198    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-140296589198656_20cf3342-2cba-4180-93a3-fad3d205b2d5/test-curl-7635-140296589198656/0.log"
Jul 17 20:52:23.839413 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.839374    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:23.839558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.839426    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l84j4\" (UniqueName: \"kubernetes.io/projected/20cf3342-2cba-4180-93a3-fad3d205b2d5-kube-api-access-l84j4\") pod \"20cf3342-2cba-4180-93a3-fad3d205b2d5\" (UID: \"20cf3342-2cba-4180-93a3-fad3d205b2d5\") "
Jul 17 20:52:23.841485 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.841462    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cf3342-2cba-4180-93a3-fad3d205b2d5-kube-api-access-l84j4" (OuterVolumeSpecName: "kube-api-access-l84j4") pod "20cf3342-2cba-4180-93a3-fad3d205b2d5" (UID: "20cf3342-2cba-4180-93a3-fad3d205b2d5"). InnerVolumeSpecName "kube-api-access-l84j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:52:23.846360 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.846332    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:23.848908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.848890    2572 scope.go:117] "RemoveContainer" containerID="054710a065be2702792fcfa22a84e7eb2456f0af2d3201fcf5fd45751862c1ad"
Jul 17 20:52:23.854721 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.854700    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20cf3342-2cba-4180-93a3-fad3d205b2d5" path="/var/lib/kubelet/pods/20cf3342-2cba-4180-93a3-fad3d205b2d5/volumes"
Jul 17 20:52:23.940795 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.940715    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84j4\" (UniqueName: \"kubernetes.io/projected/20cf3342-2cba-4180-93a3-fad3d205b2d5-kube-api-access-l84j4\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:23.961284 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.961252    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:23.961577 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.961564    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20cf3342-2cba-4180-93a3-fad3d205b2d5" containerName="test-curl-7635-140296589198656"
Jul 17 20:52:23.961621 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.961578    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="20cf3342-2cba-4180-93a3-fad3d205b2d5" containerName="test-curl-7635-140296589198656"
Jul 17 20:52:23.961662 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.961653    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="20cf3342-2cba-4180-93a3-fad3d205b2d5" containerName="test-curl-7635-140296589198656"
Jul 17 20:52:23.965749 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.965734    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:23.971065 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:23.971020    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:24.042178 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.042148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vsr\" (UniqueName: \"kubernetes.io/projected/2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521-kube-api-access-84vsr\") pod \"test-curl-7635-140296589198656\" (UID: \"2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521\") " pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:24.142870 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.142846    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vsr\" (UniqueName: \"kubernetes.io/projected/2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521-kube-api-access-84vsr\") pod \"test-curl-7635-140296589198656\" (UID: \"2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521\") " pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:24.150846 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.150822    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vsr\" (UniqueName: \"kubernetes.io/projected/2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521-kube-api-access-84vsr\") pod \"test-curl-7635-140296589198656\" (UID: \"2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521\") " pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:24.276315 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.276289    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:24.410060 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.410019    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:24.412011 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:24.411979    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b843b6b_dbd0_4a16_9f6f_fbe5a16d5521.slice/crio-4d46d2b5e41d6d28f1607999727da0245081cf589d813eacdf6b5f4902699f68 WatchSource:0}: Error finding container 4d46d2b5e41d6d28f1607999727da0245081cf589d813eacdf6b5f4902699f68: Status 404 returned error can't find the container with id 4d46d2b5e41d6d28f1607999727da0245081cf589d813eacdf6b5f4902699f68
Jul 17 20:52:24.660341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.660249    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:24.660341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.660288    2572 scope.go:117] "RemoveContainer" containerID="9c021e3215f918332c9a914f5915b096404d95bea7b5204c5ac606d982aef03c"
Jul 17 20:52:24.661614 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.661565    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589198656" event={"ID":"2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521","Type":"ContainerStarted","Data":"4d46d2b5e41d6d28f1607999727da0245081cf589d813eacdf6b5f4902699f68"}
Jul 17 20:52:24.663626 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.663599    2572 generic.go:356] "Generic (PLEG): container finished" podID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerID="f5876cbc3ba025478aa810a0c7475598fc6d992a48be6428c32ec21b273cff06" exitCode=6
Jul 17 20:52:24.663710 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.663665    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" event={"ID":"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1","Type":"ContainerDied","Data":"f5876cbc3ba025478aa810a0c7475598fc6d992a48be6428c32ec21b273cff06"}
Jul 17 20:52:24.663959 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.663941    2572 scope.go:117] "RemoveContainer" containerID="f5876cbc3ba025478aa810a0c7475598fc6d992a48be6428c32ec21b273cff06"
Jul 17 20:52:24.664150 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:24.664130    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg_odh-ai-gateway-infra(a0a222c5-fa63-4b5b-a4b3-0e9f068978a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1"
Jul 17 20:52:24.668723 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:24.668706    2572 scope.go:117] "RemoveContainer" containerID="054710a065be2702792fcfa22a84e7eb2456f0af2d3201fcf5fd45751862c1ad"
Jul 17 20:52:25.670159 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:25.670124    2572 generic.go:356] "Generic (PLEG): container finished" podID="2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521" containerID="67fe3687de174554e55418a0fd7c3c0989cffa94e3a4d182100dfd0dd154c14e" exitCode=0
Jul 17 20:52:25.670569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:25.670204    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589198656" event={"ID":"2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521","Type":"ContainerDied","Data":"67fe3687de174554e55418a0fd7c3c0989cffa94e3a4d182100dfd0dd154c14e"}
Jul 17 20:52:25.691266 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:25.691239    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"]
Jul 17 20:52:25.824201 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:25.824175    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"
Jul 17 20:52:25.853500 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:25.853472    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a222c5-fa63-4b5b-a4b3-0e9f068978a1-maas-api-service-ca\") pod \"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1\" (UID: \"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1\") "
Jul 17 20:52:25.853871 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:25.853845    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a222c5-fa63-4b5b-a4b3-0e9f068978a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" (UID: "a0a222c5-fa63-4b5b-a4b3-0e9f068978a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:25.954648 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:25.954571    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a222c5-fa63-4b5b-a4b3-0e9f068978a1-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:26.679333 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.679301    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"
Jul 17 20:52:26.679793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.679335    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg" event={"ID":"a0a222c5-fa63-4b5b-a4b3-0e9f068978a1","Type":"ContainerDied","Data":"9c7a5f9e4d745b99ad783d340a9850237362cba698d463232803312accd675c5"}
Jul 17 20:52:26.679793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.679403    2572 scope.go:117] "RemoveContainer" containerID="f5876cbc3ba025478aa810a0c7475598fc6d992a48be6428c32ec21b273cff06"
Jul 17 20:52:26.712909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.712876    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"]
Jul 17 20:52:26.721187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.721154    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-qx6qg"]
Jul 17 20:52:26.805012 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.804989    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:26.861281 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.861239    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-140296589198656_2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521/test-curl-7635-140296589198656/0.log"
Jul 17 20:52:26.898625 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.898596    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:26.909757 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.909733    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-140296589198656"]
Jul 17 20:52:26.962344 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.962313    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84vsr\" (UniqueName: \"kubernetes.io/projected/2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521-kube-api-access-84vsr\") pod \"2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521\" (UID: \"2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521\") "
Jul 17 20:52:26.964315 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:26.964293    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521-kube-api-access-84vsr" (OuterVolumeSpecName: "kube-api-access-84vsr") pod "2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521" (UID: "2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521"). InnerVolumeSpecName "kube-api-access-84vsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:52:27.053569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053531    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:27.053878 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053866    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.053936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053880    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.053936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053887    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521" containerName="test-curl-7635-140296589198656"
Jul 17 20:52:27.053936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053893    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521" containerName="test-curl-7635-140296589198656"
Jul 17 20:52:27.053936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053915    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.053936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053921    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.053936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053930    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.053936 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053937    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.054166 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.053996    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.054166 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.054005    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521" containerName="test-curl-7635-140296589198656"
Jul 17 20:52:27.054166 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.054013    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:27.056979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.056963    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:27.062862 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.062835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjcwj\" (UniqueName: \"kubernetes.io/projected/e86a2880-6099-478c-b04c-8a3c8b080967-kube-api-access-gjcwj\") pod \"test-curl-7635-140296589199664\" (UID: \"e86a2880-6099-478c-b04c-8a3c8b080967\") " pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:27.063032 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.062982    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vsr\" (UniqueName: \"kubernetes.io/projected/2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521-kube-api-access-84vsr\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:27.066231 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.066206    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:27.164034 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.163996    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjcwj\" (UniqueName: \"kubernetes.io/projected/e86a2880-6099-478c-b04c-8a3c8b080967-kube-api-access-gjcwj\") pod \"test-curl-7635-140296589199664\" (UID: \"e86a2880-6099-478c-b04c-8a3c8b080967\") " pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:27.173782 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.173753    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjcwj\" (UniqueName: \"kubernetes.io/projected/e86a2880-6099-478c-b04c-8a3c8b080967-kube-api-access-gjcwj\") pod \"test-curl-7635-140296589199664\" (UID: \"e86a2880-6099-478c-b04c-8a3c8b080967\") " pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:27.365798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.365707    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:27.505900 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.505871    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:27.507921 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:27.507889    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode86a2880_6099_478c_b04c_8a3c8b080967.slice/crio-f555ccdf619e95f77a380cf4ce8129ebadba8066d7750209fae2957c2c356dda WatchSource:0}: Error finding container f555ccdf619e95f77a380cf4ce8129ebadba8066d7750209fae2957c2c356dda: Status 404 returned error can't find the container with id f555ccdf619e95f77a380cf4ce8129ebadba8066d7750209fae2957c2c356dda
Jul 17 20:52:27.686681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.686598    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d46d2b5e41d6d28f1607999727da0245081cf589d813eacdf6b5f4902699f68"
Jul 17 20:52:27.686681 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.686603    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:27.688735 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.688708    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589199664" event={"ID":"e86a2880-6099-478c-b04c-8a3c8b080967","Type":"ContainerStarted","Data":"f555ccdf619e95f77a380cf4ce8129ebadba8066d7750209fae2957c2c356dda"}
Jul 17 20:52:27.697418 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:27.697391    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-140296589198656\" is forbidden: User \"system:node:ip-10-0-143-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-185.ec2.internal' and this object" podUID="2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521" pod="opendatahub/test-curl-7635-140296589198656"
Jul 17 20:52:27.854504 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.854473    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521" path="/var/lib/kubelet/pods/2b843b6b-dbd0-4a16-9f6f-fbe5a16d5521/volumes"
Jul 17 20:52:27.854946 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:27.854922    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" path="/var/lib/kubelet/pods/a0a222c5-fa63-4b5b-a4b3-0e9f068978a1/volumes"
Jul 17 20:52:28.695608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:28.695576    2572 generic.go:356] "Generic (PLEG): container finished" podID="e86a2880-6099-478c-b04c-8a3c8b080967" containerID="2afe9dec22414d4cd5839343d5c482ebc2b96caf6bf25fae64bff9c85077e3a1" exitCode=0
Jul 17 20:52:28.696076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:28.695668    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589199664" event={"ID":"e86a2880-6099-478c-b04c-8a3c8b080967","Type":"ContainerDied","Data":"2afe9dec22414d4cd5839343d5c482ebc2b96caf6bf25fae64bff9c85077e3a1"}
Jul 17 20:52:29.832909 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.832882    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:29.854318 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.854294    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-140296589199664_e86a2880-6099-478c-b04c-8a3c8b080967/test-curl-7635-140296589199664/0.log"
Jul 17 20:52:29.876772 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.876745    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:29.880884 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.880861    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:29.890809 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.890780    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjcwj\" (UniqueName: \"kubernetes.io/projected/e86a2880-6099-478c-b04c-8a3c8b080967-kube-api-access-gjcwj\") pod \"e86a2880-6099-478c-b04c-8a3c8b080967\" (UID: \"e86a2880-6099-478c-b04c-8a3c8b080967\") "
Jul 17 20:52:29.892855 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.892832    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86a2880-6099-478c-b04c-8a3c8b080967-kube-api-access-gjcwj" (OuterVolumeSpecName: "kube-api-access-gjcwj") pod "e86a2880-6099-478c-b04c-8a3c8b080967" (UID: "e86a2880-6099-478c-b04c-8a3c8b080967"). InnerVolumeSpecName "kube-api-access-gjcwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:52:29.992078 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.992028    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjcwj\" (UniqueName: \"kubernetes.io/projected/e86a2880-6099-478c-b04c-8a3c8b080967-kube-api-access-gjcwj\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:29.996486 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.996460    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:29.996836 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.996818    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86a2880-6099-478c-b04c-8a3c8b080967" containerName="test-curl-7635-140296589199664"
Jul 17 20:52:29.996915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.996839    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86a2880-6099-478c-b04c-8a3c8b080967" containerName="test-curl-7635-140296589199664"
Jul 17 20:52:29.996982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.996944    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86a2880-6099-478c-b04c-8a3c8b080967" containerName="test-curl-7635-140296589199664"
Jul 17 20:52:29.996982 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.996956    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a222c5-fa63-4b5b-a4b3-0e9f068978a1" containerName="revoke-keys"
Jul 17 20:52:29.999820 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:29.999800    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:30.004352 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.004326    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:30.092557 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.092525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8jcb\" (UniqueName: \"kubernetes.io/projected/7025f02f-764a-4a59-b3dd-d1a8856b0bba-kube-api-access-d8jcb\") pod \"test-curl-7635-140296589199664\" (UID: \"7025f02f-764a-4a59-b3dd-d1a8856b0bba\") " pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:30.193256 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.193209    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8jcb\" (UniqueName: \"kubernetes.io/projected/7025f02f-764a-4a59-b3dd-d1a8856b0bba-kube-api-access-d8jcb\") pod \"test-curl-7635-140296589199664\" (UID: \"7025f02f-764a-4a59-b3dd-d1a8856b0bba\") " pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:30.201673 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.201641    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8jcb\" (UniqueName: \"kubernetes.io/projected/7025f02f-764a-4a59-b3dd-d1a8856b0bba-kube-api-access-d8jcb\") pod \"test-curl-7635-140296589199664\" (UID: \"7025f02f-764a-4a59-b3dd-d1a8856b0bba\") " pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:30.308976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.308894    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:30.437356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.437330    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:30.438702 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:30.438672    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7025f02f_764a_4a59_b3dd_d1a8856b0bba.slice/crio-bdb2431fd639be62917f33eb1d4083d42a9114e1238bd7607df2ef6da8c75e21 WatchSource:0}: Error finding container bdb2431fd639be62917f33eb1d4083d42a9114e1238bd7607df2ef6da8c75e21: Status 404 returned error can't find the container with id bdb2431fd639be62917f33eb1d4083d42a9114e1238bd7607df2ef6da8c75e21
Jul 17 20:52:30.709072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.707775    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f555ccdf619e95f77a380cf4ce8129ebadba8066d7750209fae2957c2c356dda"
Jul 17 20:52:30.709072 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.707842    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:30.712326 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.712295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589199664" event={"ID":"7025f02f-764a-4a59-b3dd-d1a8856b0bba","Type":"ContainerStarted","Data":"bdb2431fd639be62917f33eb1d4083d42a9114e1238bd7607df2ef6da8c75e21"}
Jul 17 20:52:30.720427 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:30.720401    2572 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7635-140296589199664" oldPodUID="e86a2880-6099-478c-b04c-8a3c8b080967" podUID="7025f02f-764a-4a59-b3dd-d1a8856b0bba"
Jul 17 20:52:31.719445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:31.719408    2572 generic.go:356] "Generic (PLEG): container finished" podID="7025f02f-764a-4a59-b3dd-d1a8856b0bba" containerID="db1697aae32e45fa8665f7f09a3d2d878621c0366bb73c5dd83034bde26313cb" exitCode=0
Jul 17 20:52:31.719827 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:31.719502    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-140296589199664" event={"ID":"7025f02f-764a-4a59-b3dd-d1a8856b0bba","Type":"ContainerDied","Data":"db1697aae32e45fa8665f7f09a3d2d878621c0366bb73c5dd83034bde26313cb"}
Jul 17 20:52:31.853160 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:31.853131    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e86a2880-6099-478c-b04c-8a3c8b080967" path="/var/lib/kubelet/pods/e86a2880-6099-478c-b04c-8a3c8b080967/volumes"
Jul 17 20:52:31.921326 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:31.921286    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"]
Jul 17 20:52:31.924865 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:31.924846    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:52:31.927145 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:31.927125    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:52:31.931501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:31.931480    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"]
Jul 17 20:52:32.007539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.007453    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a32b92-dcc3-4a6b-80e1-faa06a4babad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-rbqvj\" (UID: \"69a32b92-dcc3-4a6b-80e1-faa06a4babad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:52:32.108015 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.107981    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a32b92-dcc3-4a6b-80e1-faa06a4babad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-rbqvj\" (UID: \"69a32b92-dcc3-4a6b-80e1-faa06a4babad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:52:32.108624 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.108602    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a32b92-dcc3-4a6b-80e1-faa06a4babad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-rbqvj\" (UID: \"69a32b92-dcc3-4a6b-80e1-faa06a4babad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:52:32.236039 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.236008    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:52:32.361562 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.361538    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"]
Jul 17 20:52:32.364172 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:32.364142    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a32b92_dcc3_4a6b_80e1_faa06a4babad.slice/crio-1391da4e51104cacd710fdcc5eb564b54ff70ba98a968caa7f65b768d6e499ae WatchSource:0}: Error finding container 1391da4e51104cacd710fdcc5eb564b54ff70ba98a968caa7f65b768d6e499ae: Status 404 returned error can't find the container with id 1391da4e51104cacd710fdcc5eb564b54ff70ba98a968caa7f65b768d6e499ae
Jul 17 20:52:32.726099 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.726038    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" event={"ID":"69a32b92-dcc3-4a6b-80e1-faa06a4babad","Type":"ContainerStarted","Data":"58fedb9e6e3bb5bc25b83e8232635a09eacd5ec716f1975a88901ba08f77d63f"}
Jul 17 20:52:32.726582 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.726122    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" event={"ID":"69a32b92-dcc3-4a6b-80e1-faa06a4babad","Type":"ContainerStarted","Data":"1391da4e51104cacd710fdcc5eb564b54ff70ba98a968caa7f65b768d6e499ae"}
Jul 17 20:52:32.743380 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.743325    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" podStartSLOduration=1.7433115670000001 podStartE2EDuration="1.743311567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:52:32.741457058 +0000 UTC m=+3387.435433218" watchObservedRunningTime="2026-07-17 20:52:32.743311567 +0000 UTC m=+3387.437287713"
Jul 17 20:52:32.853786 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.853764    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:32.875473 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.875448    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-140296589199664_7025f02f-764a-4a59-b3dd-d1a8856b0bba/test-curl-7635-140296589199664/0.log"
Jul 17 20:52:32.900597 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.900565    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:32.904934 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.904908    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-140296589199664"]
Jul 17 20:52:32.915308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.915286    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8jcb\" (UniqueName: \"kubernetes.io/projected/7025f02f-764a-4a59-b3dd-d1a8856b0bba-kube-api-access-d8jcb\") pod \"7025f02f-764a-4a59-b3dd-d1a8856b0bba\" (UID: \"7025f02f-764a-4a59-b3dd-d1a8856b0bba\") "
Jul 17 20:52:32.917336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:32.917314    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7025f02f-764a-4a59-b3dd-d1a8856b0bba-kube-api-access-d8jcb" (OuterVolumeSpecName: "kube-api-access-d8jcb") pod "7025f02f-764a-4a59-b3dd-d1a8856b0bba" (UID: "7025f02f-764a-4a59-b3dd-d1a8856b0bba"). InnerVolumeSpecName "kube-api-access-d8jcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:52:33.016008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:33.015972    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8jcb\" (UniqueName: \"kubernetes.io/projected/7025f02f-764a-4a59-b3dd-d1a8856b0bba-kube-api-access-d8jcb\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:33.732101 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:33.732069    2572 generic.go:356] "Generic (PLEG): container finished" podID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerID="58fedb9e6e3bb5bc25b83e8232635a09eacd5ec716f1975a88901ba08f77d63f" exitCode=6
Jul 17 20:52:33.732532 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:33.732149    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" event={"ID":"69a32b92-dcc3-4a6b-80e1-faa06a4babad","Type":"ContainerDied","Data":"58fedb9e6e3bb5bc25b83e8232635a09eacd5ec716f1975a88901ba08f77d63f"}
Jul 17 20:52:33.732532 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:33.732334    2572 scope.go:117] "RemoveContainer" containerID="58fedb9e6e3bb5bc25b83e8232635a09eacd5ec716f1975a88901ba08f77d63f"
Jul 17 20:52:33.733424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:33.733406    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb2431fd639be62917f33eb1d4083d42a9114e1238bd7607df2ef6da8c75e21"
Jul 17 20:52:33.733424 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:33.733417    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:33.750739 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:33.750705    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-140296589199664\" is forbidden: User \"system:node:ip-10-0-143-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-185.ec2.internal' and this object" podUID="7025f02f-764a-4a59-b3dd-d1a8856b0bba" pod="opendatahub/test-curl-7635-140296589199664"
Jul 17 20:52:33.853903 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:33.853870    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7025f02f-764a-4a59-b3dd-d1a8856b0bba" path="/var/lib/kubelet/pods/7025f02f-764a-4a59-b3dd-d1a8856b0bba/volumes"
Jul 17 20:52:34.740213 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:34.740175    2572 generic.go:356] "Generic (PLEG): container finished" podID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerID="20516a50d7d8010d00525a6e368f2e6a8990fb319cf2cf65a3480961b8c485e6" exitCode=6
Jul 17 20:52:34.740664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:34.740220    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" event={"ID":"69a32b92-dcc3-4a6b-80e1-faa06a4babad","Type":"ContainerDied","Data":"20516a50d7d8010d00525a6e368f2e6a8990fb319cf2cf65a3480961b8c485e6"}
Jul 17 20:52:34.740664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:34.740269    2572 scope.go:117] "RemoveContainer" containerID="58fedb9e6e3bb5bc25b83e8232635a09eacd5ec716f1975a88901ba08f77d63f"
Jul 17 20:52:34.740664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:34.740462    2572 scope.go:117] "RemoveContainer" containerID="20516a50d7d8010d00525a6e368f2e6a8990fb319cf2cf65a3480961b8c485e6"
Jul 17 20:52:34.740801 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:34.740668    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-rbqvj_odh-ai-gateway-infra(69a32b92-dcc3-4a6b-80e1-faa06a4babad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad"
Jul 17 20:52:35.746586 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:35.746555    2572 scope.go:117] "RemoveContainer" containerID="20516a50d7d8010d00525a6e368f2e6a8990fb319cf2cf65a3480961b8c485e6"
Jul 17 20:52:35.746977 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:35.746728    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-rbqvj_odh-ai-gateway-infra(69a32b92-dcc3-4a6b-80e1-faa06a4babad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad"
Jul 17 20:52:36.024748 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.024656    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"]
Jul 17 20:52:36.025095 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.025078    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7025f02f-764a-4a59-b3dd-d1a8856b0bba" containerName="test-curl-7635-140296589199664"
Jul 17 20:52:36.025194 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.025097    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7025f02f-764a-4a59-b3dd-d1a8856b0bba" containerName="test-curl-7635-140296589199664"
Jul 17 20:52:36.025194 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.025181    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7025f02f-764a-4a59-b3dd-d1a8856b0bba" containerName="test-curl-7635-140296589199664"
Jul 17 20:52:36.029366 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.029345    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"
Jul 17 20:52:36.040994 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.040966    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"]
Jul 17 20:52:36.141976 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.141939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/087493d4-3b6a-494d-b0bd-d3518e0c7165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-8b22q\" (UID: \"087493d4-3b6a-494d-b0bd-d3518e0c7165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"
Jul 17 20:52:36.243345 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.243308    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/087493d4-3b6a-494d-b0bd-d3518e0c7165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-8b22q\" (UID: \"087493d4-3b6a-494d-b0bd-d3518e0c7165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"
Jul 17 20:52:36.243987 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.243959    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/087493d4-3b6a-494d-b0bd-d3518e0c7165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-8b22q\" (UID: \"087493d4-3b6a-494d-b0bd-d3518e0c7165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"
Jul 17 20:52:36.345699 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.345613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"
Jul 17 20:52:36.680634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.680610    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"]
Jul 17 20:52:36.682659 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:36.682634    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087493d4_3b6a_494d_b0bd_d3518e0c7165.slice/crio-9a28dc319f6118122082ed32d74611972ad39bb2697adb6b8a020b8321707fcb WatchSource:0}: Error finding container 9a28dc319f6118122082ed32d74611972ad39bb2697adb6b8a020b8321707fcb: Status 404 returned error can't find the container with id 9a28dc319f6118122082ed32d74611972ad39bb2697adb6b8a020b8321707fcb
Jul 17 20:52:36.752301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:36.752268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" event={"ID":"087493d4-3b6a-494d-b0bd-d3518e0c7165","Type":"ContainerStarted","Data":"9a28dc319f6118122082ed32d74611972ad39bb2697adb6b8a020b8321707fcb"}
Jul 17 20:52:37.759224 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:37.759189    2572 generic.go:356] "Generic (PLEG): container finished" podID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerID="aba1d8a930fd43ad8c1d2f6562faa670f639bd4582cd498cbc0511c59a93671a" exitCode=6
Jul 17 20:52:37.759601 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:37.759263    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" event={"ID":"087493d4-3b6a-494d-b0bd-d3518e0c7165","Type":"ContainerDied","Data":"aba1d8a930fd43ad8c1d2f6562faa670f639bd4582cd498cbc0511c59a93671a"}
Jul 17 20:52:37.759601 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:37.759434    2572 scope.go:117] "RemoveContainer" containerID="aba1d8a930fd43ad8c1d2f6562faa670f639bd4582cd498cbc0511c59a93671a"
Jul 17 20:52:38.242583 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.242545    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"]
Jul 17 20:52:38.245930 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.245908    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"
Jul 17 20:52:38.253404 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.253370    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"]
Jul 17 20:52:38.360921 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.360884    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb\" (UID: \"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"
Jul 17 20:52:38.462258 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.462223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb\" (UID: \"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"
Jul 17 20:52:38.462832 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.462814    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb\" (UID: \"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"
Jul 17 20:52:38.557808 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.557734    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"
Jul 17 20:52:38.687821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.687787    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"]
Jul 17 20:52:38.691041 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:38.691017    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c92e09_37f1_44a1_9a5b_f9d2ad63e8ac.slice/crio-bf7de9d1b8131d46e3cba878967a90569059d556e82a20a930906ce0338e6ee9 WatchSource:0}: Error finding container bf7de9d1b8131d46e3cba878967a90569059d556e82a20a930906ce0338e6ee9: Status 404 returned error can't find the container with id bf7de9d1b8131d46e3cba878967a90569059d556e82a20a930906ce0338e6ee9
Jul 17 20:52:38.769029 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.768988    2572 generic.go:356] "Generic (PLEG): container finished" podID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerID="f5f419ecd88e243e54fa31fd5fa3c0a36b3ad5c37ca0f5be916792f2c6e0464e" exitCode=6
Jul 17 20:52:38.769514 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.769072    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" event={"ID":"087493d4-3b6a-494d-b0bd-d3518e0c7165","Type":"ContainerDied","Data":"f5f419ecd88e243e54fa31fd5fa3c0a36b3ad5c37ca0f5be916792f2c6e0464e"}
Jul 17 20:52:38.769514 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.769129    2572 scope.go:117] "RemoveContainer" containerID="aba1d8a930fd43ad8c1d2f6562faa670f639bd4582cd498cbc0511c59a93671a"
Jul 17 20:52:38.769514 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.769315    2572 scope.go:117] "RemoveContainer" containerID="f5f419ecd88e243e54fa31fd5fa3c0a36b3ad5c37ca0f5be916792f2c6e0464e"
Jul 17 20:52:38.769677 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:38.769560    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-8b22q_odh-ai-gateway-infra(087493d4-3b6a-494d-b0bd-d3518e0c7165)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165"
Jul 17 20:52:38.770632 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:38.770609    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" event={"ID":"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac","Type":"ContainerStarted","Data":"bf7de9d1b8131d46e3cba878967a90569059d556e82a20a930906ce0338e6ee9"}
Jul 17 20:52:39.776769 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:39.776739    2572 generic.go:356] "Generic (PLEG): container finished" podID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerID="05844ad341f5bd6573ec9c1e8ecbe4cb85aa5e431f7ab51ba406ec67d3027080" exitCode=6
Jul 17 20:52:39.777211 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:39.776823    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" event={"ID":"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac","Type":"ContainerDied","Data":"05844ad341f5bd6573ec9c1e8ecbe4cb85aa5e431f7ab51ba406ec67d3027080"}
Jul 17 20:52:39.777211 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:39.776975    2572 scope.go:117] "RemoveContainer" containerID="05844ad341f5bd6573ec9c1e8ecbe4cb85aa5e431f7ab51ba406ec67d3027080"
Jul 17 20:52:39.778724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:39.778709    2572 scope.go:117] "RemoveContainer" containerID="f5f419ecd88e243e54fa31fd5fa3c0a36b3ad5c37ca0f5be916792f2c6e0464e"
Jul 17 20:52:39.778870 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:39.778852    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-8b22q_odh-ai-gateway-infra(087493d4-3b6a-494d-b0bd-d3518e0c7165)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165"
Jul 17 20:52:40.784691 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:40.784656    2572 generic.go:356] "Generic (PLEG): container finished" podID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerID="0cfb6e8fafce4ebd035badde9716f9b5863caecfa511e47ffde918ce9aced3ad" exitCode=6
Jul 17 20:52:40.785114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:40.784737    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" event={"ID":"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac","Type":"ContainerDied","Data":"0cfb6e8fafce4ebd035badde9716f9b5863caecfa511e47ffde918ce9aced3ad"}
Jul 17 20:52:40.785114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:40.784789    2572 scope.go:117] "RemoveContainer" containerID="05844ad341f5bd6573ec9c1e8ecbe4cb85aa5e431f7ab51ba406ec67d3027080"
Jul 17 20:52:40.785114 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:40.785009    2572 scope.go:117] "RemoveContainer" containerID="0cfb6e8fafce4ebd035badde9716f9b5863caecfa511e47ffde918ce9aced3ad"
Jul 17 20:52:40.785250 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:40.785233    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb_odh-ai-gateway-infra(05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac"
Jul 17 20:52:41.791501 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:41.791472    2572 scope.go:117] "RemoveContainer" containerID="0cfb6e8fafce4ebd035badde9716f9b5863caecfa511e47ffde918ce9aced3ad"
Jul 17 20:52:41.791881 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:41.791643    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb_odh-ai-gateway-infra(05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac"
Jul 17 20:52:43.630821 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:43.630786    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"]
Jul 17 20:52:43.631301 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:43.631028    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf" podUID="ea81aa58-21b7-4b5a-b309-d31497116f65" containerName="maas-api" containerID="cri-o://0ed2ec00b5f02ed978b826685e8bda83aa45f68b210ef3dfdbfaa4cdf217cab4" gracePeriod=30
Jul 17 20:52:43.803454 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:43.803418    2572 generic.go:356] "Generic (PLEG): container finished" podID="ea81aa58-21b7-4b5a-b309-d31497116f65" containerID="0ed2ec00b5f02ed978b826685e8bda83aa45f68b210ef3dfdbfaa4cdf217cab4" exitCode=0
Jul 17 20:52:43.803612 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:43.803581    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf" event={"ID":"ea81aa58-21b7-4b5a-b309-d31497116f65","Type":"ContainerDied","Data":"0ed2ec00b5f02ed978b826685e8bda83aa45f68b210ef3dfdbfaa4cdf217cab4"}
Jul 17 20:52:43.883683 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:43.883625    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:52:44.012265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.012227    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f67f\" (UniqueName: \"kubernetes.io/projected/ea81aa58-21b7-4b5a-b309-d31497116f65-kube-api-access-6f67f\") pod \"ea81aa58-21b7-4b5a-b309-d31497116f65\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") "
Jul 17 20:52:44.012445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.012325    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea81aa58-21b7-4b5a-b309-d31497116f65-openshift-service-ca\") pod \"ea81aa58-21b7-4b5a-b309-d31497116f65\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") "
Jul 17 20:52:44.012445 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.012385    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls\") pod \"ea81aa58-21b7-4b5a-b309-d31497116f65\" (UID: \"ea81aa58-21b7-4b5a-b309-d31497116f65\") "
Jul 17 20:52:44.012698 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.012667    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea81aa58-21b7-4b5a-b309-d31497116f65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea81aa58-21b7-4b5a-b309-d31497116f65" (UID: "ea81aa58-21b7-4b5a-b309-d31497116f65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:44.014422 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.014393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea81aa58-21b7-4b5a-b309-d31497116f65" (UID: "ea81aa58-21b7-4b5a-b309-d31497116f65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 20:52:44.014422 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.014406    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea81aa58-21b7-4b5a-b309-d31497116f65-kube-api-access-6f67f" (OuterVolumeSpecName: "kube-api-access-6f67f") pod "ea81aa58-21b7-4b5a-b309-d31497116f65" (UID: "ea81aa58-21b7-4b5a-b309-d31497116f65"). InnerVolumeSpecName "kube-api-access-6f67f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:52:44.113103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.113072    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea81aa58-21b7-4b5a-b309-d31497116f65-openshift-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:44.113103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.113099    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea81aa58-21b7-4b5a-b309-d31497116f65-maas-api-tls\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:44.113103 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.113108    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f67f\" (UniqueName: \"kubernetes.io/projected/ea81aa58-21b7-4b5a-b309-d31497116f65-kube-api-access-6f67f\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:44.809589 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.809557    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf" event={"ID":"ea81aa58-21b7-4b5a-b309-d31497116f65","Type":"ContainerDied","Data":"c3478620c6f3f8e54a8c025c74b4a10cd6fca34754133d3aaded44a866d69892"}
Jul 17 20:52:44.809589 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.809579    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"
Jul 17 20:52:44.810089 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.809618    2572 scope.go:117] "RemoveContainer" containerID="0ed2ec00b5f02ed978b826685e8bda83aa45f68b210ef3dfdbfaa4cdf217cab4"
Jul 17 20:52:44.831341 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.831315    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"]
Jul 17 20:52:44.834942 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:44.834919    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72eefe-69557666c4-kjgqf"]
Jul 17 20:52:45.853326 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:45.853290    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea81aa58-21b7-4b5a-b309-d31497116f65" path="/var/lib/kubelet/pods/ea81aa58-21b7-4b5a-b309-d31497116f65/volumes"
Jul 17 20:52:47.145895 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:47.145864    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:52:48.848724 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:48.848696    2572 scope.go:117] "RemoveContainer" containerID="20516a50d7d8010d00525a6e368f2e6a8990fb319cf2cf65a3480961b8c485e6"
Jul 17 20:52:49.838309 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:49.838280    2572 generic.go:356] "Generic (PLEG): container finished" podID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerID="e06f0dcb9599a7b3c25c744fcb43024436a266b5a0c4ec96414765ea20ab28bc" exitCode=6
Jul 17 20:52:49.838464 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:49.838354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" event={"ID":"69a32b92-dcc3-4a6b-80e1-faa06a4babad","Type":"ContainerDied","Data":"e06f0dcb9599a7b3c25c744fcb43024436a266b5a0c4ec96414765ea20ab28bc"}
Jul 17 20:52:49.838464 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:49.838417    2572 scope.go:117] "RemoveContainer" containerID="20516a50d7d8010d00525a6e368f2e6a8990fb319cf2cf65a3480961b8c485e6"
Jul 17 20:52:49.838707 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:49.838687    2572 scope.go:117] "RemoveContainer" containerID="e06f0dcb9599a7b3c25c744fcb43024436a266b5a0c4ec96414765ea20ab28bc"
Jul 17 20:52:49.838955 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:49.838903    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-rbqvj_odh-ai-gateway-infra(69a32b92-dcc3-4a6b-80e1-faa06a4babad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad"
Jul 17 20:52:50.847851 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:50.847827    2572 scope.go:117] "RemoveContainer" containerID="f5f419ecd88e243e54fa31fd5fa3c0a36b3ad5c37ca0f5be916792f2c6e0464e"
Jul 17 20:52:50.868755 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:50.868727    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"]
Jul 17 20:52:51.051793 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.051769    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:52:51.178640 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.178604    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a32b92-dcc3-4a6b-80e1-faa06a4babad-maas-api-service-ca\") pod \"69a32b92-dcc3-4a6b-80e1-faa06a4babad\" (UID: \"69a32b92-dcc3-4a6b-80e1-faa06a4babad\") "
Jul 17 20:52:51.178921 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.178898    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a32b92-dcc3-4a6b-80e1-faa06a4babad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69a32b92-dcc3-4a6b-80e1-faa06a4babad" (UID: "69a32b92-dcc3-4a6b-80e1-faa06a4babad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:51.279704 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.279674    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a32b92-dcc3-4a6b-80e1-faa06a4babad-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:51.851547 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.851455    2572 generic.go:356] "Generic (PLEG): container finished" podID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerID="d22d2757f1c71da07b557b63ff1db7e3f5c8438ac950e4185feff80bf723c4b6" exitCode=6
Jul 17 20:52:51.852764 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.852743    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:52:51.854309 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.854274    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" event={"ID":"087493d4-3b6a-494d-b0bd-d3518e0c7165","Type":"ContainerDied","Data":"d22d2757f1c71da07b557b63ff1db7e3f5c8438ac950e4185feff80bf723c4b6"}
Jul 17 20:52:51.854431 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.854333    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" event={"ID":"69a32b92-dcc3-4a6b-80e1-faa06a4babad","Type":"ContainerDied","Data":"1391da4e51104cacd710fdcc5eb564b54ff70ba98a968caa7f65b768d6e499ae"}
Jul 17 20:52:51.854431 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.854370    2572 scope.go:117] "RemoveContainer" containerID="f5f419ecd88e243e54fa31fd5fa3c0a36b3ad5c37ca0f5be916792f2c6e0464e"
Jul 17 20:52:51.854656 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.854630    2572 scope.go:117] "RemoveContainer" containerID="d22d2757f1c71da07b557b63ff1db7e3f5c8438ac950e4185feff80bf723c4b6"
Jul 17 20:52:51.854868 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:51.854843    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-8b22q_odh-ai-gateway-infra(087493d4-3b6a-494d-b0bd-d3518e0c7165)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165"
Jul 17 20:52:51.866675 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:51.866659    2572 scope.go:117] "RemoveContainer" containerID="e06f0dcb9599a7b3c25c744fcb43024436a266b5a0c4ec96414765ea20ab28bc"
Jul 17 20:52:52.887347 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:52.887310    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"]
Jul 17 20:52:53.020317 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.020291    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"
Jul 17 20:52:53.095455 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.095420    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/087493d4-3b6a-494d-b0bd-d3518e0c7165-maas-api-service-ca\") pod \"087493d4-3b6a-494d-b0bd-d3518e0c7165\" (UID: \"087493d4-3b6a-494d-b0bd-d3518e0c7165\") "
Jul 17 20:52:53.095766 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.095744    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087493d4-3b6a-494d-b0bd-d3518e0c7165-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "087493d4-3b6a-494d-b0bd-d3518e0c7165" (UID: "087493d4-3b6a-494d-b0bd-d3518e0c7165"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:53.196634 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.196554    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/087493d4-3b6a-494d-b0bd-d3518e0c7165-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:53.865355 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.865331    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"
Jul 17 20:52:53.865493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.865368    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q" event={"ID":"087493d4-3b6a-494d-b0bd-d3518e0c7165","Type":"ContainerDied","Data":"9a28dc319f6118122082ed32d74611972ad39bb2697adb6b8a020b8321707fcb"}
Jul 17 20:52:53.865493 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.865428    2572 scope.go:117] "RemoveContainer" containerID="d22d2757f1c71da07b557b63ff1db7e3f5c8438ac950e4185feff80bf723c4b6"
Jul 17 20:52:53.885085 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.885039    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"]
Jul 17 20:52:53.891595 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:53.891574    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-8b22q"]
Jul 17 20:52:55.850429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:55.850390    2572 scope.go:117] "RemoveContainer" containerID="0cfb6e8fafce4ebd035badde9716f9b5863caecfa511e47ffde918ce9aced3ad"
Jul 17 20:52:55.853740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:55.853712    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" path="/var/lib/kubelet/pods/087493d4-3b6a-494d-b0bd-d3518e0c7165/volumes"
Jul 17 20:52:56.881361 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:56.881268    2572 generic.go:356] "Generic (PLEG): container finished" podID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerID="05b91cb44e4269d01f469cedf7837eb749240028231e90cdff9424fdfc010e1f" exitCode=6
Jul 17 20:52:56.881361 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:56.881345    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" event={"ID":"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac","Type":"ContainerDied","Data":"05b91cb44e4269d01f469cedf7837eb749240028231e90cdff9424fdfc010e1f"}
Jul 17 20:52:56.881817 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:56.881404    2572 scope.go:117] "RemoveContainer" containerID="0cfb6e8fafce4ebd035badde9716f9b5863caecfa511e47ffde918ce9aced3ad"
Jul 17 20:52:56.881817 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:56.881678    2572 scope.go:117] "RemoveContainer" containerID="05b91cb44e4269d01f469cedf7837eb749240028231e90cdff9424fdfc010e1f"
Jul 17 20:52:56.881904 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:52:56.881874    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb_odh-ai-gateway-infra(05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac"
Jul 17 20:52:57.912797 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:57.912726    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"]
Jul 17 20:52:58.045472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.045448    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"
Jul 17 20:52:58.141570 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.141535    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac-maas-api-service-ca\") pod \"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac\" (UID: \"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac\") "
Jul 17 20:52:58.141892 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.141870    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" (UID: "05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:52:58.243024 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.242996    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:52:58.894231 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.894201    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"
Jul 17 20:52:58.894410 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.894200    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb" event={"ID":"05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac","Type":"ContainerDied","Data":"bf7de9d1b8131d46e3cba878967a90569059d556e82a20a930906ce0338e6ee9"}
Jul 17 20:52:58.894410 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.894336    2572 scope.go:117] "RemoveContainer" containerID="05b91cb44e4269d01f469cedf7837eb749240028231e90cdff9424fdfc010e1f"
Jul 17 20:52:58.916013 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.915990    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"]
Jul 17 20:52:58.918983 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:58.918960    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-ktgrb"]
Jul 17 20:52:59.269678 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269642    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"]
Jul 17 20:52:59.269951 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269940    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269952    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269961    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269966    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269976    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269982    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269989    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.269994    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.269998 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270000    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270005    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270018    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270023    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270031    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270036    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270041    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea81aa58-21b7-4b5a-b309-d31497116f65" containerName="maas-api"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270058    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea81aa58-21b7-4b5a-b309-d31497116f65" containerName="maas-api"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270105    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270112    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270118    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270124    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea81aa58-21b7-4b5a-b309-d31497116f65" containerName="maas-api"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270129    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270135    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270140    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="087493d4-3b6a-494d-b0bd-d3518e0c7165" containerName="revoke-keys"
Jul 17 20:52:59.270259 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.270146    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:52:59.274407 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.274390    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"
Jul 17 20:52:59.277312 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.277291    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:52:59.294082 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.294041    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"]
Jul 17 20:52:59.456513 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.456468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a22450-9aa9-4ad8-ab38-4a1cad6687f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-327614-849g4\" (UID: \"67a22450-9aa9-4ad8-ab38-4a1cad6687f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"
Jul 17 20:52:59.558009 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.557920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a22450-9aa9-4ad8-ab38-4a1cad6687f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-327614-849g4\" (UID: \"67a22450-9aa9-4ad8-ab38-4a1cad6687f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"
Jul 17 20:52:59.558551 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.558531    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a22450-9aa9-4ad8-ab38-4a1cad6687f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-327614-849g4\" (UID: \"67a22450-9aa9-4ad8-ab38-4a1cad6687f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"
Jul 17 20:52:59.583989 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.583962    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"
Jul 17 20:52:59.730534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.730502    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"]
Jul 17 20:52:59.732803 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:52:59.732775    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a22450_9aa9_4ad8_ab38_4a1cad6687f1.slice/crio-584a68218bca2dfd28bc39f29ab1e5f43d2c21f8d2fd68544e29faaa1314df47 WatchSource:0}: Error finding container 584a68218bca2dfd28bc39f29ab1e5f43d2c21f8d2fd68544e29faaa1314df47: Status 404 returned error can't find the container with id 584a68218bca2dfd28bc39f29ab1e5f43d2c21f8d2fd68544e29faaa1314df47
Jul 17 20:52:59.853294 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.853266    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" path="/var/lib/kubelet/pods/05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac/volumes"
Jul 17 20:52:59.900399 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:52:59.900361    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4" event={"ID":"67a22450-9aa9-4ad8-ab38-4a1cad6687f1","Type":"ContainerStarted","Data":"584a68218bca2dfd28bc39f29ab1e5f43d2c21f8d2fd68544e29faaa1314df47"}
Jul 17 20:53:00.068674 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.068644    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"]
Jul 17 20:53:00.069090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.068959    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:53:00.069090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.068969    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:53:00.069090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.068979    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:53:00.069090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.068984    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:53:00.069090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.069030    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c92e09-37f1-44a1-9a5b-f9d2ad63e8ac" containerName="revoke-keys"
Jul 17 20:53:00.069090 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.069039    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" containerName="revoke-keys"
Jul 17 20:53:00.071965 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.071939    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"
Jul 17 20:53:00.081777 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.081754    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"]
Jul 17 20:53:00.162946 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.162851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860184ea-e021-4bc7-8784-c376fe39a877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw\" (UID: \"860184ea-e021-4bc7-8784-c376fe39a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"
Jul 17 20:53:00.264248 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.264218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860184ea-e021-4bc7-8784-c376fe39a877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw\" (UID: \"860184ea-e021-4bc7-8784-c376fe39a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"
Jul 17 20:53:00.264819 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.264798    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860184ea-e021-4bc7-8784-c376fe39a877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw\" (UID: \"860184ea-e021-4bc7-8784-c376fe39a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"
Jul 17 20:53:00.382729 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.382697    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"
Jul 17 20:53:00.512784 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.512723    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"]
Jul 17 20:53:00.514838 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:53:00.514812    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860184ea_e021_4bc7_8784_c376fe39a877.slice/crio-6295a8a2e0789b51048a70470ae09b198d8404a3b2df0634eca20c849e4a43e6 WatchSource:0}: Error finding container 6295a8a2e0789b51048a70470ae09b198d8404a3b2df0634eca20c849e4a43e6: Status 404 returned error can't find the container with id 6295a8a2e0789b51048a70470ae09b198d8404a3b2df0634eca20c849e4a43e6
Jul 17 20:53:00.909273 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.909232    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" event={"ID":"860184ea-e021-4bc7-8784-c376fe39a877","Type":"ContainerStarted","Data":"3ed7e565b55b6167c7ab001ac88d630d69a7c31798482aa13f1caf317ed10a36"}
Jul 17 20:53:00.909465 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.909296    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" event={"ID":"860184ea-e021-4bc7-8784-c376fe39a877","Type":"ContainerStarted","Data":"6295a8a2e0789b51048a70470ae09b198d8404a3b2df0634eca20c849e4a43e6"}
Jul 17 20:53:00.910652 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.910625    2572 generic.go:356] "Generic (PLEG): container finished" podID="67a22450-9aa9-4ad8-ab38-4a1cad6687f1" containerID="1f1961cef79bb8c7153f71e3e10e8240b8d894cc06526e5a327e3c583e81c839" exitCode=0
Jul 17 20:53:00.910763 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.910658    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4" event={"ID":"67a22450-9aa9-4ad8-ab38-4a1cad6687f1","Type":"ContainerDied","Data":"1f1961cef79bb8c7153f71e3e10e8240b8d894cc06526e5a327e3c583e81c839"}
Jul 17 20:53:00.924539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:00.924488    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" podStartSLOduration=0.924469736 podStartE2EDuration="924.469736ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:53:00.923662411 +0000 UTC m=+3415.617638584" watchObservedRunningTime="2026-07-17 20:53:00.924469736 +0000 UTC m=+3415.618445885"
Jul 17 20:53:01.917633 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:01.917598    2572 generic.go:356] "Generic (PLEG): container finished" podID="860184ea-e021-4bc7-8784-c376fe39a877" containerID="3ed7e565b55b6167c7ab001ac88d630d69a7c31798482aa13f1caf317ed10a36" exitCode=6
Jul 17 20:53:01.918146 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:01.917678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" event={"ID":"860184ea-e021-4bc7-8784-c376fe39a877","Type":"ContainerDied","Data":"3ed7e565b55b6167c7ab001ac88d630d69a7c31798482aa13f1caf317ed10a36"}
Jul 17 20:53:01.918146 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:01.917864    2572 scope.go:117] "RemoveContainer" containerID="3ed7e565b55b6167c7ab001ac88d630d69a7c31798482aa13f1caf317ed10a36"
Jul 17 20:53:02.049277 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.049257    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"
Jul 17 20:53:02.185033 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.184964    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a22450-9aa9-4ad8-ab38-4a1cad6687f1-maas-api-service-ca\") pod \"67a22450-9aa9-4ad8-ab38-4a1cad6687f1\" (UID: \"67a22450-9aa9-4ad8-ab38-4a1cad6687f1\") "
Jul 17 20:53:02.185288 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.185269    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a22450-9aa9-4ad8-ab38-4a1cad6687f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67a22450-9aa9-4ad8-ab38-4a1cad6687f1" (UID: "67a22450-9aa9-4ad8-ab38-4a1cad6687f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:53:02.286308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.286278    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a22450-9aa9-4ad8-ab38-4a1cad6687f1-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:02.924293 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.924209    2572 generic.go:356] "Generic (PLEG): container finished" podID="860184ea-e021-4bc7-8784-c376fe39a877" containerID="e10a1128ff33295323eb40d0d351a19ea4ada5f75661281f4e3166ed772a989c" exitCode=6
Jul 17 20:53:02.924719 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.924303    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" event={"ID":"860184ea-e021-4bc7-8784-c376fe39a877","Type":"ContainerDied","Data":"e10a1128ff33295323eb40d0d351a19ea4ada5f75661281f4e3166ed772a989c"}
Jul 17 20:53:02.924719 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.924370    2572 scope.go:117] "RemoveContainer" containerID="3ed7e565b55b6167c7ab001ac88d630d69a7c31798482aa13f1caf317ed10a36"
Jul 17 20:53:02.924719 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.924486    2572 scope.go:117] "RemoveContainer" containerID="e10a1128ff33295323eb40d0d351a19ea4ada5f75661281f4e3166ed772a989c"
Jul 17 20:53:02.924911 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:02.924728    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw_odh-ai-gateway-infra(860184ea-e021-4bc7-8784-c376fe39a877)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" podUID="860184ea-e021-4bc7-8784-c376fe39a877"
Jul 17 20:53:02.925734 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.925712    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4" event={"ID":"67a22450-9aa9-4ad8-ab38-4a1cad6687f1","Type":"ContainerDied","Data":"584a68218bca2dfd28bc39f29ab1e5f43d2c21f8d2fd68544e29faaa1314df47"}
Jul 17 20:53:02.925734 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.925728    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-327614-849g4"
Jul 17 20:53:02.925840 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:02.925756    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="584a68218bca2dfd28bc39f29ab1e5f43d2c21f8d2fd68544e29faaa1314df47"
Jul 17 20:53:03.932242 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:03.932213    2572 scope.go:117] "RemoveContainer" containerID="e10a1128ff33295323eb40d0d351a19ea4ada5f75661281f4e3166ed772a989c"
Jul 17 20:53:03.932720 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:03.932436    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw_odh-ai-gateway-infra(860184ea-e021-4bc7-8784-c376fe39a877)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" podUID="860184ea-e021-4bc7-8784-c376fe39a877"
Jul 17 20:53:09.878013 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:09.877982    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"]
Jul 17 20:53:09.878530 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:09.878246    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw" podUID="60cfcabc-83c2-42d0-bea0-0ef980bd3c69" containerName="maas-api" containerID="cri-o://a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc" gracePeriod=30
Jul 17 20:53:10.127404 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.127378    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:53:10.142151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.142076    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-openshift-service-ca\") pod \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") "
Jul 17 20:53:10.142151 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.142133    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drqzf\" (UniqueName: \"kubernetes.io/projected/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-kube-api-access-drqzf\") pod \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") "
Jul 17 20:53:10.142344 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.142161    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls\") pod \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\" (UID: \"60cfcabc-83c2-42d0-bea0-0ef980bd3c69\") "
Jul 17 20:53:10.142498 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.142474    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60cfcabc-83c2-42d0-bea0-0ef980bd3c69" (UID: "60cfcabc-83c2-42d0-bea0-0ef980bd3c69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:53:10.144860 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.144817    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-kube-api-access-drqzf" (OuterVolumeSpecName: "kube-api-access-drqzf") pod "60cfcabc-83c2-42d0-bea0-0ef980bd3c69" (UID: "60cfcabc-83c2-42d0-bea0-0ef980bd3c69"). InnerVolumeSpecName "kube-api-access-drqzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 20:53:10.145517 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.145479    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60cfcabc-83c2-42d0-bea0-0ef980bd3c69" (UID: "60cfcabc-83c2-42d0-bea0-0ef980bd3c69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 20:53:10.243429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.243395    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drqzf\" (UniqueName: \"kubernetes.io/projected/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-kube-api-access-drqzf\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:10.243429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.243423    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-maas-api-tls\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:10.243429 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.243435    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfcabc-83c2-42d0-bea0-0ef980bd3c69-openshift-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:10.964558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.964524    2572 generic.go:356] "Generic (PLEG): container finished" podID="60cfcabc-83c2-42d0-bea0-0ef980bd3c69" containerID="a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc" exitCode=0
Jul 17 20:53:10.965076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.964565    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"
Jul 17 20:53:10.965076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.964593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw" event={"ID":"60cfcabc-83c2-42d0-bea0-0ef980bd3c69","Type":"ContainerDied","Data":"a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc"}
Jul 17 20:53:10.965076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.964643    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw" event={"ID":"60cfcabc-83c2-42d0-bea0-0ef980bd3c69","Type":"ContainerDied","Data":"981b45460deea2cd654a3e3d715c73c59ed9fb414acd006376653fe17b675d83"}
Jul 17 20:53:10.965076 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.964675    2572 scope.go:117] "RemoveContainer" containerID="a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc"
Jul 17 20:53:10.973939 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.973913    2572 scope.go:117] "RemoveContainer" containerID="a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc"
Jul 17 20:53:10.974248 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:10.974225    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc\": container with ID starting with a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc not found: ID does not exist" containerID="a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc"
Jul 17 20:53:10.974371 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.974258    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc"} err="failed to get container status \"a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc\": rpc error: code = NotFound desc = could not find container \"a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc\": container with ID starting with a674d5ab0c2952dcd40bb6043675fd3e023ecff98f500e36e9ea36203d4558cc not found: ID does not exist"
Jul 17 20:53:10.984749 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.984727    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"]
Jul 17 20:53:10.991136 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:10.991115    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-327614-96565676b-mj2rw"]
Jul 17 20:53:11.853062 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:11.853015    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60cfcabc-83c2-42d0-bea0-0ef980bd3c69" path="/var/lib/kubelet/pods/60cfcabc-83c2-42d0-bea0-0ef980bd3c69/volumes"
Jul 17 20:53:12.888798 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:12.888766    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-csfcv"]
Jul 17 20:53:17.848003 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:17.847974    2572 scope.go:117] "RemoveContainer" containerID="e10a1128ff33295323eb40d0d351a19ea4ada5f75661281f4e3166ed772a989c"
Jul 17 20:53:19.003740 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:19.003710    2572 generic.go:356] "Generic (PLEG): container finished" podID="860184ea-e021-4bc7-8784-c376fe39a877" containerID="eaea5511a7ff591765146d9433e7d065136bba3996943b92a00924987982dbc1" exitCode=6
Jul 17 20:53:19.004188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:19.003768    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" event={"ID":"860184ea-e021-4bc7-8784-c376fe39a877","Type":"ContainerDied","Data":"eaea5511a7ff591765146d9433e7d065136bba3996943b92a00924987982dbc1"}
Jul 17 20:53:19.004188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:19.003816    2572 scope.go:117] "RemoveContainer" containerID="e10a1128ff33295323eb40d0d351a19ea4ada5f75661281f4e3166ed772a989c"
Jul 17 20:53:19.004188 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:19.004100    2572 scope.go:117] "RemoveContainer" containerID="eaea5511a7ff591765146d9433e7d065136bba3996943b92a00924987982dbc1"
Jul 17 20:53:19.004355 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:19.004287    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw_odh-ai-gateway-infra(860184ea-e021-4bc7-8784-c376fe39a877)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" podUID="860184ea-e021-4bc7-8784-c376fe39a877"
Jul 17 20:53:20.030900 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:20.030865    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"]
Jul 17 20:53:20.164484 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:20.164462    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"
Jul 17 20:53:20.218671 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:20.218636    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860184ea-e021-4bc7-8784-c376fe39a877-maas-api-service-ca\") pod \"860184ea-e021-4bc7-8784-c376fe39a877\" (UID: \"860184ea-e021-4bc7-8784-c376fe39a877\") "
Jul 17 20:53:20.219017 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:20.218987    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860184ea-e021-4bc7-8784-c376fe39a877-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "860184ea-e021-4bc7-8784-c376fe39a877" (UID: "860184ea-e021-4bc7-8784-c376fe39a877"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:53:20.319682 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:20.319606    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860184ea-e021-4bc7-8784-c376fe39a877-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:21.015744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.015712    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw" event={"ID":"860184ea-e021-4bc7-8784-c376fe39a877","Type":"ContainerDied","Data":"6295a8a2e0789b51048a70470ae09b198d8404a3b2df0634eca20c849e4a43e6"}
Jul 17 20:53:21.015744 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.015746    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"
Jul 17 20:53:21.015966 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.015770    2572 scope.go:117] "RemoveContainer" containerID="eaea5511a7ff591765146d9433e7d065136bba3996943b92a00924987982dbc1"
Jul 17 20:53:21.037187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.037149    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"]
Jul 17 20:53:21.043320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.043298    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-rmlhw"]
Jul 17 20:53:21.853618 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.853581    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="860184ea-e021-4bc7-8784-c376fe39a877" path="/var/lib/kubelet/pods/860184ea-e021-4bc7-8784-c376fe39a877/volumes"
Jul 17 20:53:21.862850 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.862815    2572 pod_container_manager_linux.go:217] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod69a32b92-dcc3-4a6b-80e1-faa06a4babad"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod69a32b92-dcc3-4a6b-80e1-faa06a4babad] : Timed out while waiting for systemd to remove kubepods-burstable-pod69a32b92_dcc3_4a6b_80e1_faa06a4babad.slice"
Jul 17 20:53:21.863005 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:21.862853    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod69a32b92-dcc3-4a6b-80e1-faa06a4babad] : unable to destroy cgroup paths for cgroup [kubepods burstable pod69a32b92-dcc3-4a6b-80e1-faa06a4babad] : Timed out while waiting for systemd to remove kubepods-burstable-pod69a32b92_dcc3_4a6b_80e1_faa06a4babad.slice" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad"
Jul 17 20:53:21.970756 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.970724    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"]
Jul 17 20:53:21.971025 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971013    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a22450-9aa9-4ad8-ab38-4a1cad6687f1" containerName="revoke-keys"
Jul 17 20:53:21.971111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971026    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a22450-9aa9-4ad8-ab38-4a1cad6687f1" containerName="revoke-keys"
Jul 17 20:53:21.971111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971035    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:21.971111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971040    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:21.971111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971074    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60cfcabc-83c2-42d0-bea0-0ef980bd3c69" containerName="maas-api"
Jul 17 20:53:21.971111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971083    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="60cfcabc-83c2-42d0-bea0-0ef980bd3c69" containerName="maas-api"
Jul 17 20:53:21.971111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971093    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:21.971111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971103    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:21.971353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971156    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:21.971353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971163    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="60cfcabc-83c2-42d0-bea0-0ef980bd3c69" containerName="maas-api"
Jul 17 20:53:21.971353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971169    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:21.971353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971175    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:21.971353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.971182    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a22450-9aa9-4ad8-ab38-4a1cad6687f1" containerName="revoke-keys"
Jul 17 20:53:21.975357 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.975341    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"
Jul 17 20:53:21.978002 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.977982    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:53:21.978149 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.978126    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:53:21.981210 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:21.981191    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"]
Jul 17 20:53:22.021111 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.021084    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"
Jul 17 20:53:22.034058 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.034028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf5822d-8073-4cb6-b72e-a302f97c43d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-s4ct9\" (UID: \"1bf5822d-8073-4cb6-b72e-a302f97c43d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"
Jul 17 20:53:22.051871 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.051846    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"]
Jul 17 20:53:22.058068 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.058035    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-rbqvj"]
Jul 17 20:53:22.134880 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.134791    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf5822d-8073-4cb6-b72e-a302f97c43d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-s4ct9\" (UID: \"1bf5822d-8073-4cb6-b72e-a302f97c43d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"
Jul 17 20:53:22.135504 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.135474    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf5822d-8073-4cb6-b72e-a302f97c43d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5420d171-s4ct9\" (UID: \"1bf5822d-8073-4cb6-b72e-a302f97c43d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"
Jul 17 20:53:22.286579 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.286546    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"
Jul 17 20:53:22.419560 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:22.419486    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"]
Jul 17 20:53:22.422612 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:53:22.422582    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf5822d_8073_4cb6_b72e_a302f97c43d1.slice/crio-8780cc8c901e4210c4d2ea4f889efe819d0d8254d33dab7d2b5d0e7816b731dc WatchSource:0}: Error finding container 8780cc8c901e4210c4d2ea4f889efe819d0d8254d33dab7d2b5d0e7816b731dc: Status 404 returned error can't find the container with id 8780cc8c901e4210c4d2ea4f889efe819d0d8254d33dab7d2b5d0e7816b731dc
Jul 17 20:53:23.029155 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:23.029008    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" event={"ID":"1bf5822d-8073-4cb6-b72e-a302f97c43d1","Type":"ContainerStarted","Data":"fa60c20ac36fb08169ff01cbd4275d41659673ad376cd734834113200f2e80c5"}
Jul 17 20:53:23.029307 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:23.029194    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" event={"ID":"1bf5822d-8073-4cb6-b72e-a302f97c43d1","Type":"ContainerStarted","Data":"8780cc8c901e4210c4d2ea4f889efe819d0d8254d33dab7d2b5d0e7816b731dc"}
Jul 17 20:53:23.048271 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:23.048227    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" podStartSLOduration=2.048212929 podStartE2EDuration="2.048212929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:53:23.046295035 +0000 UTC m=+3437.740271181" watchObservedRunningTime="2026-07-17 20:53:23.048212929 +0000 UTC m=+3437.742189075"
Jul 17 20:53:23.853952 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:23.853918    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a32b92-dcc3-4a6b-80e1-faa06a4babad" path="/var/lib/kubelet/pods/69a32b92-dcc3-4a6b-80e1-faa06a4babad/volumes"
Jul 17 20:53:24.036012 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:24.035973    2572 generic.go:356] "Generic (PLEG): container finished" podID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerID="fa60c20ac36fb08169ff01cbd4275d41659673ad376cd734834113200f2e80c5" exitCode=6
Jul 17 20:53:24.036187 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:24.036074    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" event={"ID":"1bf5822d-8073-4cb6-b72e-a302f97c43d1","Type":"ContainerDied","Data":"fa60c20ac36fb08169ff01cbd4275d41659673ad376cd734834113200f2e80c5"}
Jul 17 20:53:24.036265 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:24.036251    2572 scope.go:117] "RemoveContainer" containerID="fa60c20ac36fb08169ff01cbd4275d41659673ad376cd734834113200f2e80c5"
Jul 17 20:53:25.043631 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:25.043593    2572 generic.go:356] "Generic (PLEG): container finished" podID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerID="649a9092c6aa939f66b14b659f8337d133f92b0228640e6ed0440aa6bc795c32" exitCode=6
Jul 17 20:53:25.044120 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:25.043676    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" event={"ID":"1bf5822d-8073-4cb6-b72e-a302f97c43d1","Type":"ContainerDied","Data":"649a9092c6aa939f66b14b659f8337d133f92b0228640e6ed0440aa6bc795c32"}
Jul 17 20:53:25.044120 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:25.043744    2572 scope.go:117] "RemoveContainer" containerID="fa60c20ac36fb08169ff01cbd4275d41659673ad376cd734834113200f2e80c5"
Jul 17 20:53:25.044120 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:25.043942    2572 scope.go:117] "RemoveContainer" containerID="649a9092c6aa939f66b14b659f8337d133f92b0228640e6ed0440aa6bc795c32"
Jul 17 20:53:25.044255 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:25.044193    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-s4ct9_odh-ai-gateway-infra(1bf5822d-8073-4cb6-b72e-a302f97c43d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1"
Jul 17 20:53:26.050356 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.050330    2572 scope.go:117] "RemoveContainer" containerID="649a9092c6aa939f66b14b659f8337d133f92b0228640e6ed0440aa6bc795c32"
Jul 17 20:53:26.050764 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:26.050512    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-s4ct9_odh-ai-gateway-infra(1bf5822d-8073-4cb6-b72e-a302f97c43d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1"
Jul 17 20:53:26.075509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.075479    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"]
Jul 17 20:53:26.075802 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.075790    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:26.075847 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.075803    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="860184ea-e021-4bc7-8784-c376fe39a877" containerName="revoke-keys"
Jul 17 20:53:26.080033 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.080016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"
Jul 17 20:53:26.091649 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.091625    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"]
Jul 17 20:53:26.167651 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.167612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/218141cf-1cc5-4e48-8e70-4e9bc0b4d857-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw\" (UID: \"218141cf-1cc5-4e48-8e70-4e9bc0b4d857\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"
Jul 17 20:53:26.269030 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.268988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/218141cf-1cc5-4e48-8e70-4e9bc0b4d857-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw\" (UID: \"218141cf-1cc5-4e48-8e70-4e9bc0b4d857\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"
Jul 17 20:53:26.269623 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.269604    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/218141cf-1cc5-4e48-8e70-4e9bc0b4d857-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw\" (UID: \"218141cf-1cc5-4e48-8e70-4e9bc0b4d857\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"
Jul 17 20:53:26.388960 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.388877    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"
Jul 17 20:53:26.525421 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:26.525397    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"]
Jul 17 20:53:26.528397 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:53:26.528367    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218141cf_1cc5_4e48_8e70_4e9bc0b4d857.slice/crio-40366c56cd7bdfe6d17951312d314555c67ac05023f582f5e98f25e046095a51 WatchSource:0}: Error finding container 40366c56cd7bdfe6d17951312d314555c67ac05023f582f5e98f25e046095a51: Status 404 returned error can't find the container with id 40366c56cd7bdfe6d17951312d314555c67ac05023f582f5e98f25e046095a51
Jul 17 20:53:27.056816 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:27.056721    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" event={"ID":"218141cf-1cc5-4e48-8e70-4e9bc0b4d857","Type":"ContainerStarted","Data":"9aca1f0fedf9e7a9edc715a9f9ccc53530b40db586d2777670becc6c93ab1797"}
Jul 17 20:53:27.056816 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:27.056804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" event={"ID":"218141cf-1cc5-4e48-8e70-4e9bc0b4d857","Type":"ContainerStarted","Data":"40366c56cd7bdfe6d17951312d314555c67ac05023f582f5e98f25e046095a51"}
Jul 17 20:53:27.082177 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:27.082121    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" podStartSLOduration=1.082103897 podStartE2EDuration="1.082103897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:53:27.08091902 +0000 UTC m=+3441.774895167" watchObservedRunningTime="2026-07-17 20:53:27.082103897 +0000 UTC m=+3441.776080044"
Jul 17 20:53:28.063215 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:28.063180    2572 generic.go:356] "Generic (PLEG): container finished" podID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerID="9aca1f0fedf9e7a9edc715a9f9ccc53530b40db586d2777670becc6c93ab1797" exitCode=6
Jul 17 20:53:28.063613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:28.063268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" event={"ID":"218141cf-1cc5-4e48-8e70-4e9bc0b4d857","Type":"ContainerDied","Data":"9aca1f0fedf9e7a9edc715a9f9ccc53530b40db586d2777670becc6c93ab1797"}
Jul 17 20:53:28.063613 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:28.063408    2572 scope.go:117] "RemoveContainer" containerID="9aca1f0fedf9e7a9edc715a9f9ccc53530b40db586d2777670becc6c93ab1797"
Jul 17 20:53:29.069214 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:29.069178    2572 generic.go:356] "Generic (PLEG): container finished" podID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerID="685694590d86a44983156e1baca58237445b2b3530ce4e314a0ac79851bbff34" exitCode=6
Jul 17 20:53:29.069639 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:29.069240    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" event={"ID":"218141cf-1cc5-4e48-8e70-4e9bc0b4d857","Type":"ContainerDied","Data":"685694590d86a44983156e1baca58237445b2b3530ce4e314a0ac79851bbff34"}
Jul 17 20:53:29.069639 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:29.069303    2572 scope.go:117] "RemoveContainer" containerID="9aca1f0fedf9e7a9edc715a9f9ccc53530b40db586d2777670becc6c93ab1797"
Jul 17 20:53:29.069639 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:29.069516    2572 scope.go:117] "RemoveContainer" containerID="685694590d86a44983156e1baca58237445b2b3530ce4e314a0ac79851bbff34"
Jul 17 20:53:29.069777 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:29.069734    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw_odh-ai-gateway-infra(218141cf-1cc5-4e48-8e70-4e9bc0b4d857)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857"
Jul 17 20:53:30.075244 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:30.075220    2572 scope.go:117] "RemoveContainer" containerID="685694590d86a44983156e1baca58237445b2b3530ce4e314a0ac79851bbff34"
Jul 17 20:53:30.075735 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:30.075458    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw_odh-ai-gateway-infra(218141cf-1cc5-4e48-8e70-4e9bc0b4d857)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857"
Jul 17 20:53:30.876617 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:30.876588    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s5fvl_5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9/manager/0.log"
Jul 17 20:53:30.991008 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:30.990974    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bxc4f_e9666413-1a40-4ad3-b2bc-113de3eb747f/manager/0.log"
Jul 17 20:53:31.217901 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:31.217879    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nhgzn_8e526790-634c-4370-8785-fa57241af7e9/server/0.log"
Jul 17 20:53:31.330286 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:31.330256    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mfjbf_ad0bb1e5-caf8-40cb-a9b8-7013d746e028/manager/0.log"
Jul 17 20:53:31.442381 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:31.442356    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm_74343d95-b829-4a9d-a2c5-35b7dc48c6b4/copy-manifests/0.log"
Jul 17 20:53:31.454458 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:31.454437    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm_74343d95-b829-4a9d-a2c5-35b7dc48c6b4/manager/0.log"
Jul 17 20:53:32.167215 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.167185    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738655-5zrj9_1be5be68-aa82-462e-9ab1-59506b52b1c0/cleanup/0.log"
Jul 17 20:53:32.273149 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.273120    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738670-nspnc_8f320eb5-cac2-48a5-a8bc-3c829238e69c/cleanup/0.log"
Jul 17 20:53:32.382656 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.382630    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738685-fxbx6_76d6b614-00c8-46b1-baaa-0a5a159bdeed/cleanup/0.log"
Jul 17 20:53:32.489143 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.489117    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-5420d171-s4ct9_1bf5822d-8073-4cb6-b72e-a302f97c43d1/revoke-keys/1.log"
Jul 17 20:53:32.610832 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.610803    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-3a4159-8pkjm_5931c3b3-e453-448d-905c-5879a4a08812/revoke-keys/0.log"
Jul 17 20:53:32.722581 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.722552    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-ad0e32-w8vfk_3b56ed59-c174-47e1-80ae-626557b7e990/revoke-keys/0.log"
Jul 17 20:53:32.835999 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.835924    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw_218141cf-1cc5-4e48-8e70-4e9bc0b4d857/revoke-keys/1.log"
Jul 17 20:53:32.941010 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:32.940982    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-13e7d9-7vsfv_35b4aabc-fa84-42fa-8810-b4c91bcd6675/revoke-keys/0.log"
Jul 17 20:53:33.059680 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:33.059643    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-5f0f4f-vtphc_d72f75b7-d1ee-4a0f-90d3-1d872af07c72/revoke-keys/0.log"
Jul 17 20:53:33.166219 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:33.166141    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-2c0bc539-tmj4k_ca2fd047-a1e1-400a-83ac-78d5bc5b3d7f/revoke-keys/0.log"
Jul 17 20:53:33.273027 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:33.272995    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-6d55d2b6-whwtr_d685b191-086f-4bd9-8ea9-800e8e073a17/revoke-keys/0.log"
Jul 17 20:53:33.509620 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:33.509591    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-327614-849g4_67a22450-9aa9-4ad8-ab38-4a1cad6687f1/revoke-keys/0.log"
Jul 17 20:53:35.421295 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:35.421266    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6xkvq_87dbe9f7-be3a-45d4-b8a9-0437ab606d7a/manager/0.log"
Jul 17 20:53:35.529369 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:35.529340    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-csfcv_52b478c2-e1c8-47e5-b742-55a254ff683c/limitador/0.log"
Jul 17 20:53:35.645422 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:35.645394    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bj95b_ee60f502-c989-4a07-8561-d62e6c81bb7f/manager/0.log"
Jul 17 20:53:36.097527 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:36.097497    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mfnl8_dcfa7f00-ae9c-4e06-a14d-937beeaa6381/discovery/0.log"
Jul 17 20:53:36.206509 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:36.206469    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fdbd865b7-2zgvv_be6d4ef4-252f-4dd5-9570-3eacd013e3b3/kube-auth-proxy/0.log"
Jul 17 20:53:38.302145 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.302110    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"]
Jul 17 20:53:38.306666 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.306649    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"
Jul 17 20:53:38.313870 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.313363    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"]
Jul 17 20:53:38.359334 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.359302    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b97cff-db42-4920-838c-39efd272e517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-669mw\" (UID: \"98b97cff-db42-4920-838c-39efd272e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"
Jul 17 20:53:38.460726 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.460691    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b97cff-db42-4920-838c-39efd272e517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-669mw\" (UID: \"98b97cff-db42-4920-838c-39efd272e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"
Jul 17 20:53:38.461305 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.461287    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b97cff-db42-4920-838c-39efd272e517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9a08062a-669mw\" (UID: \"98b97cff-db42-4920-838c-39efd272e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"
Jul 17 20:53:38.618496 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.618397    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"
Jul 17 20:53:38.751168 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:38.750327    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"]
Jul 17 20:53:38.752632 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:53:38.752596    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b97cff_db42_4920_838c_39efd272e517.slice/crio-b6825e608d7d8e72a41df19a25cdeec65e4a8da6413a9fca6f6fd2654532d1da WatchSource:0}: Error finding container b6825e608d7d8e72a41df19a25cdeec65e4a8da6413a9fca6f6fd2654532d1da: Status 404 returned error can't find the container with id b6825e608d7d8e72a41df19a25cdeec65e4a8da6413a9fca6f6fd2654532d1da
Jul 17 20:53:39.120467 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:39.120435    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" event={"ID":"98b97cff-db42-4920-838c-39efd272e517","Type":"ContainerStarted","Data":"bb6b185dbd4e563a113b38124ecc139ae661a743711c723b5bc09fbf22434386"}
Jul 17 20:53:39.120653 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:39.120490    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" event={"ID":"98b97cff-db42-4920-838c-39efd272e517","Type":"ContainerStarted","Data":"b6825e608d7d8e72a41df19a25cdeec65e4a8da6413a9fca6f6fd2654532d1da"}
Jul 17 20:53:39.140121 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:39.140078    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" podStartSLOduration=1.140063756 podStartE2EDuration="1.140063756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:53:39.138102044 +0000 UTC m=+3453.832078186" watchObservedRunningTime="2026-07-17 20:53:39.140063756 +0000 UTC m=+3453.834039896"
Jul 17 20:53:40.126815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:40.126780    2572 generic.go:356] "Generic (PLEG): container finished" podID="98b97cff-db42-4920-838c-39efd272e517" containerID="bb6b185dbd4e563a113b38124ecc139ae661a743711c723b5bc09fbf22434386" exitCode=6
Jul 17 20:53:40.127242 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:40.126854    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" event={"ID":"98b97cff-db42-4920-838c-39efd272e517","Type":"ContainerDied","Data":"bb6b185dbd4e563a113b38124ecc139ae661a743711c723b5bc09fbf22434386"}
Jul 17 20:53:40.127242 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:40.127004    2572 scope.go:117] "RemoveContainer" containerID="bb6b185dbd4e563a113b38124ecc139ae661a743711c723b5bc09fbf22434386"
Jul 17 20:53:40.848633 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:40.848612    2572 scope.go:117] "RemoveContainer" containerID="649a9092c6aa939f66b14b659f8337d133f92b0228640e6ed0440aa6bc795c32"
Jul 17 20:53:40.848739 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:40.848725    2572 scope.go:117] "RemoveContainer" containerID="685694590d86a44983156e1baca58237445b2b3530ce4e314a0ac79851bbff34"
Jul 17 20:53:41.133472 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:41.133443    2572 generic.go:356] "Generic (PLEG): container finished" podID="98b97cff-db42-4920-838c-39efd272e517" containerID="0161f2458537c8a4933f0899251a97d7ae3c4d8621451e61709c594359d6a203" exitCode=6
Jul 17 20:53:41.133830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:41.133523    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" event={"ID":"98b97cff-db42-4920-838c-39efd272e517","Type":"ContainerDied","Data":"0161f2458537c8a4933f0899251a97d7ae3c4d8621451e61709c594359d6a203"}
Jul 17 20:53:41.133830 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:41.133598    2572 scope.go:117] "RemoveContainer" containerID="bb6b185dbd4e563a113b38124ecc139ae661a743711c723b5bc09fbf22434386"
Jul 17 20:53:41.134034 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:41.133829    2572 scope.go:117] "RemoveContainer" containerID="0161f2458537c8a4933f0899251a97d7ae3c4d8621451e61709c594359d6a203"
Jul 17 20:53:41.134125 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:41.134034    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-669mw_odh-ai-gateway-infra(98b97cff-db42-4920-838c-39efd272e517)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" podUID="98b97cff-db42-4920-838c-39efd272e517"
Jul 17 20:53:42.139885 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.139856    2572 generic.go:356] "Generic (PLEG): container finished" podID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerID="aab123882bb44a3ad126d8fcbb2ea0f3ee0dd4603c646e55b23741425dd1e4ce" exitCode=6
Jul 17 20:53:42.140340 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.139925    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" event={"ID":"218141cf-1cc5-4e48-8e70-4e9bc0b4d857","Type":"ContainerDied","Data":"aab123882bb44a3ad126d8fcbb2ea0f3ee0dd4603c646e55b23741425dd1e4ce"}
Jul 17 20:53:42.140340 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.139983    2572 scope.go:117] "RemoveContainer" containerID="685694590d86a44983156e1baca58237445b2b3530ce4e314a0ac79851bbff34"
Jul 17 20:53:42.140474 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.140352    2572 scope.go:117] "RemoveContainer" containerID="aab123882bb44a3ad126d8fcbb2ea0f3ee0dd4603c646e55b23741425dd1e4ce"
Jul 17 20:53:42.140595 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:42.140570    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw_odh-ai-gateway-infra(218141cf-1cc5-4e48-8e70-4e9bc0b4d857)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857"
Jul 17 20:53:42.141820 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.141801    2572 generic.go:356] "Generic (PLEG): container finished" podID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerID="446434fce76173d9b911dac8da95d0ee30bf2f728a30c76bd5ce74d871d8fad7" exitCode=6
Jul 17 20:53:42.141903 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.141871    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" event={"ID":"1bf5822d-8073-4cb6-b72e-a302f97c43d1","Type":"ContainerDied","Data":"446434fce76173d9b911dac8da95d0ee30bf2f728a30c76bd5ce74d871d8fad7"}
Jul 17 20:53:42.142167 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.142144    2572 scope.go:117] "RemoveContainer" containerID="446434fce76173d9b911dac8da95d0ee30bf2f728a30c76bd5ce74d871d8fad7"
Jul 17 20:53:42.142334 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:42.142317    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5420d171-s4ct9_odh-ai-gateway-infra(1bf5822d-8073-4cb6-b72e-a302f97c43d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1"
Jul 17 20:53:42.143534 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.143519    2572 scope.go:117] "RemoveContainer" containerID="0161f2458537c8a4933f0899251a97d7ae3c4d8621451e61709c594359d6a203"
Jul 17 20:53:42.143692 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:42.143674    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-669mw_odh-ai-gateway-infra(98b97cff-db42-4920-838c-39efd272e517)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" podUID="98b97cff-db42-4920-838c-39efd272e517"
Jul 17 20:53:42.150804 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:42.150790    2572 scope.go:117] "RemoveContainer" containerID="649a9092c6aa939f66b14b659f8337d133f92b0228640e6ed0440aa6bc795c32"
Jul 17 20:53:43.165827 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.165798    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"]
Jul 17 20:53:43.187146 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.187118    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"]
Jul 17 20:53:43.308721 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.308699    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"
Jul 17 20:53:43.346606 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.346582    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"
Jul 17 20:53:43.402528 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.402487    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf5822d-8073-4cb6-b72e-a302f97c43d1-maas-api-service-ca\") pod \"1bf5822d-8073-4cb6-b72e-a302f97c43d1\" (UID: \"1bf5822d-8073-4cb6-b72e-a302f97c43d1\") "
Jul 17 20:53:43.402688 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.402577    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/218141cf-1cc5-4e48-8e70-4e9bc0b4d857-maas-api-service-ca\") pod \"218141cf-1cc5-4e48-8e70-4e9bc0b4d857\" (UID: \"218141cf-1cc5-4e48-8e70-4e9bc0b4d857\") "
Jul 17 20:53:43.402870 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.402850    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf5822d-8073-4cb6-b72e-a302f97c43d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1bf5822d-8073-4cb6-b72e-a302f97c43d1" (UID: "1bf5822d-8073-4cb6-b72e-a302f97c43d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:53:43.402910 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.402886    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218141cf-1cc5-4e48-8e70-4e9bc0b4d857-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "218141cf-1cc5-4e48-8e70-4e9bc0b4d857" (UID: "218141cf-1cc5-4e48-8e70-4e9bc0b4d857"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:53:43.503663 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.503631    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/218141cf-1cc5-4e48-8e70-4e9bc0b4d857-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:43.503663 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:43.503658    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf5822d-8073-4cb6-b72e-a302f97c43d1-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:44.163529 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.163454    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"
Jul 17 20:53:44.163529 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.163500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9" event={"ID":"1bf5822d-8073-4cb6-b72e-a302f97c43d1","Type":"ContainerDied","Data":"8780cc8c901e4210c4d2ea4f889efe819d0d8254d33dab7d2b5d0e7816b731dc"}
Jul 17 20:53:44.163776 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.163554    2572 scope.go:117] "RemoveContainer" containerID="446434fce76173d9b911dac8da95d0ee30bf2f728a30c76bd5ce74d871d8fad7"
Jul 17 20:53:44.165000 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.164976    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw" event={"ID":"218141cf-1cc5-4e48-8e70-4e9bc0b4d857","Type":"ContainerDied","Data":"40366c56cd7bdfe6d17951312d314555c67ac05023f582f5e98f25e046095a51"}
Jul 17 20:53:44.165132 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.164992    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"
Jul 17 20:53:44.172794 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.172575    2572 scope.go:117] "RemoveContainer" containerID="aab123882bb44a3ad126d8fcbb2ea0f3ee0dd4603c646e55b23741425dd1e4ce"
Jul 17 20:53:44.182979 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.182958    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"]
Jul 17 20:53:44.188912 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.188882    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5420d171-s4ct9"]
Jul 17 20:53:44.200401 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.200375    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"]
Jul 17 20:53:44.206664 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:44.206640    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1a20513-j5nmw"]
Jul 17 20:53:45.853007 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:45.852975    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" path="/var/lib/kubelet/pods/1bf5822d-8073-4cb6-b72e-a302f97c43d1/volumes"
Jul 17 20:53:45.853400 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:45.853306    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" path="/var/lib/kubelet/pods/218141cf-1cc5-4e48-8e70-4e9bc0b4d857/volumes"
Jul 17 20:53:46.271067 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:46.271023    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mdlgw_c642abb8-6e39-49db-955a-442707a3f0a6/global-pull-secret-syncer/0.log"
Jul 17 20:53:46.477512 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:46.477484    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zxldh_be9e287a-187f-49b2-aafd-75205fcdff77/konnectivity-agent/0.log"
Jul 17 20:53:46.546317 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:46.546238    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-185.ec2.internal_fee423c54644a9085fb43b1dbbd566c9/haproxy/0.log"
Jul 17 20:53:51.374140 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:51.374104    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6xkvq_87dbe9f7-be3a-45d4-b8a9-0437ab606d7a/manager/0.log"
Jul 17 20:53:51.395234 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:51.395211    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-csfcv_52b478c2-e1c8-47e5-b742-55a254ff683c/limitador/0.log"
Jul 17 20:53:51.474968 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:51.474931    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bj95b_ee60f502-c989-4a07-8561-d62e6c81bb7f/manager/0.log"
Jul 17 20:53:53.409105 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:53.409066    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pt9xq_20597f50-0946-417f-a0d3-a76a765a9572/node-exporter/0.log"
Jul 17 20:53:53.433260 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:53.433233    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pt9xq_20597f50-0946-417f-a0d3-a76a765a9572/kube-rbac-proxy/0.log"
Jul 17 20:53:53.456752 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:53.456728    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pt9xq_20597f50-0946-417f-a0d3-a76a765a9572/init-textfile/0.log"
Jul 17 20:53:54.006915 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:54.006882    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-wmzqn_da4702ca-10a0-405c-b98d-41a8642b57c4/prometheus-operator/0.log"
Jul 17 20:53:54.027041 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:54.027020    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-wmzqn_da4702ca-10a0-405c-b98d-41a8642b57c4/kube-rbac-proxy/0.log"
Jul 17 20:53:55.851352 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:55.851324    2572 scope.go:117] "RemoveContainer" containerID="0161f2458537c8a4933f0899251a97d7ae3c4d8621451e61709c594359d6a203"
Jul 17 20:53:56.226180 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:56.226147    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" event={"ID":"98b97cff-db42-4920-838c-39efd272e517","Type":"ContainerStarted","Data":"0b555492e667fcb723121b29588e4366a1855f398b89caf84754e7ff2ce89b7e"}
Jul 17 20:53:57.232701 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.232664    2572 generic.go:356] "Generic (PLEG): container finished" podID="98b97cff-db42-4920-838c-39efd272e517" containerID="0b555492e667fcb723121b29588e4366a1855f398b89caf84754e7ff2ce89b7e" exitCode=6
Jul 17 20:53:57.233177 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.232736    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" event={"ID":"98b97cff-db42-4920-838c-39efd272e517","Type":"ContainerDied","Data":"0b555492e667fcb723121b29588e4366a1855f398b89caf84754e7ff2ce89b7e"}
Jul 17 20:53:57.233177 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.232818    2572 scope.go:117] "RemoveContainer" containerID="0161f2458537c8a4933f0899251a97d7ae3c4d8621451e61709c594359d6a203"
Jul 17 20:53:57.233177 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.233137    2572 scope.go:117] "RemoveContainer" containerID="0b555492e667fcb723121b29588e4366a1855f398b89caf84754e7ff2ce89b7e"
Jul 17 20:53:57.233399 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:53:57.233378    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9a08062a-669mw_odh-ai-gateway-infra(98b97cff-db42-4920-838c-39efd272e517)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" podUID="98b97cff-db42-4920-838c-39efd272e517"
Jul 17 20:53:57.272254 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272224    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"]
Jul 17 20:53:57.272558 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272547    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:53:57.272602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272560    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:53:57.272602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272568    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:53:57.272602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272574    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:53:57.272602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272584    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:53:57.272602 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272589    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:53:57.272762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272607    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:53:57.272762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272613    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:53:57.272762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272665    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:53:57.272762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272674    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:53:57.272762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272680    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:53:57.272762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272685    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:53:57.272762 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.272691    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:53:57.276807 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.276788    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.281353 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.281334    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlh9g\"/\"openshift-service-ca.crt\""
Jul 17 20:53:57.283539 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.283513    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlh9g\"/\"kube-root-ca.crt\""
Jul 17 20:53:57.283768 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.283750    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"]
Jul 17 20:53:57.284435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.284417    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wlh9g\"/\"default-dockercfg-kgm6k\""
Jul 17 20:53:57.341336 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.341303    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"]
Jul 17 20:53:57.418304 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.418268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-sys\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.418304 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.418307    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xtj\" (UniqueName: \"kubernetes.io/projected/eba49c29-fa95-45d7-a19c-b24a3b3da177-kube-api-access-p2xtj\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.418543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.418332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-podres\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.418543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.418381    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-proc\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.418543 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.418439    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-lib-modules\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.519869 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.519779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-lib-modules\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.519869 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.519844    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-sys\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.520109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.519901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-sys\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.520109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.519948    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xtj\" (UniqueName: \"kubernetes.io/projected/eba49c29-fa95-45d7-a19c-b24a3b3da177-kube-api-access-p2xtj\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.520109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.519972    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-lib-modules\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.520109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.519999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-podres\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.520109 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.520022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-proc\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.520367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.520139    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-proc\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.520367 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.520144    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eba49c29-fa95-45d7-a19c-b24a3b3da177-podres\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.532141 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.532109    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xtj\" (UniqueName: \"kubernetes.io/projected/eba49c29-fa95-45d7-a19c-b24a3b3da177-kube-api-access-p2xtj\") pod \"perf-node-gather-daemonset-ff6h7\" (UID: \"eba49c29-fa95-45d7-a19c-b24a3b3da177\") " pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.586765 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.586701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:57.735939 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:57.735916    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"]
Jul 17 20:53:57.738406 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:53:57.738366    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeba49c29_fa95_45d7_a19c_b24a3b3da177.slice/crio-fdf2fd64a77ae0a8dc802e4c2930448bc426fd412599c47f106c2098fc8521d2 WatchSource:0}: Error finding container fdf2fd64a77ae0a8dc802e4c2930448bc426fd412599c47f106c2098fc8521d2: Status 404 returned error can't find the container with id fdf2fd64a77ae0a8dc802e4c2930448bc426fd412599c47f106c2098fc8521d2
Jul 17 20:53:58.241115 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.241086    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7" event={"ID":"eba49c29-fa95-45d7-a19c-b24a3b3da177","Type":"ContainerStarted","Data":"9cc0631c86092e0407b9a6576c1b82bf9229bd2cb0112c9b4bd359c3ca8fe9f6"}
Jul 17 20:53:58.241482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.241149    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:53:58.241482 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.241181    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7" event={"ID":"eba49c29-fa95-45d7-a19c-b24a3b3da177","Type":"ContainerStarted","Data":"fdf2fd64a77ae0a8dc802e4c2930448bc426fd412599c47f106c2098fc8521d2"}
Jul 17 20:53:58.275231 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.275178    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7" podStartSLOduration=1.2751635590000001 podStartE2EDuration="1.275163559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 20:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:53:58.273352819 +0000 UTC m=+3472.967328966" watchObservedRunningTime="2026-07-17 20:53:58.275163559 +0000 UTC m=+3472.969139741"
Jul 17 20:53:58.366698 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.366674    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"
Jul 17 20:53:58.528272 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.528187    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b97cff-db42-4920-838c-39efd272e517-maas-api-service-ca\") pod \"98b97cff-db42-4920-838c-39efd272e517\" (UID: \"98b97cff-db42-4920-838c-39efd272e517\") "
Jul 17 20:53:58.528555 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.528532    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b97cff-db42-4920-838c-39efd272e517-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98b97cff-db42-4920-838c-39efd272e517" (UID: "98b97cff-db42-4920-838c-39efd272e517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 20:53:58.628670 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:58.628639    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b97cff-db42-4920-838c-39efd272e517-maas-api-service-ca\") on node \"ip-10-0-143-185.ec2.internal\" DevicePath \"\""
Jul 17 20:53:59.008127 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.008103    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9n9c2_cf391883-f0be-4ab2-a2dd-309cd902f61b/dns/0.log"
Jul 17 20:53:59.039391 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.039359    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9n9c2_cf391883-f0be-4ab2-a2dd-309cd902f61b/kube-rbac-proxy/0.log"
Jul 17 20:53:59.073820 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.073795    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5z8z9_031d2f1f-e63a-4d02-ac4e-bd7a4eca7773/dns-node-resolver/0.log"
Jul 17 20:53:59.246815 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.246788    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"
Jul 17 20:53:59.247296 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.246825    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw" event={"ID":"98b97cff-db42-4920-838c-39efd272e517","Type":"ContainerDied","Data":"b6825e608d7d8e72a41df19a25cdeec65e4a8da6413a9fca6f6fd2654532d1da"}
Jul 17 20:53:59.247296 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.246889    2572 scope.go:117] "RemoveContainer" containerID="0b555492e667fcb723121b29588e4366a1855f398b89caf84754e7ff2ce89b7e"
Jul 17 20:53:59.285080 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.284996    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"]
Jul 17 20:53:59.290159 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.290137    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9a08062a-669mw"]
Jul 17 20:53:59.853186 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.853155    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b97cff-db42-4920-838c-39efd272e517" path="/var/lib/kubelet/pods/98b97cff-db42-4920-838c-39efd272e517/volumes"
Jul 17 20:53:59.944419 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:53:59.944384    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pjbwc_1e158eee-6883-41ae-996a-5dad5e07629b/node-ca/0.log"
Jul 17 20:54:00.124516 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124438    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"]
Jul 17 20:54:00.124753 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124742    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b97cff-db42-4920-838c-39efd272e517" containerName="revoke-keys"
Jul 17 20:54:00.124805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124755    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b97cff-db42-4920-838c-39efd272e517" containerName="revoke-keys"
Jul 17 20:54:00.124805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124762    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:54:00.124805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124768    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:54:00.124805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124780    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b97cff-db42-4920-838c-39efd272e517" containerName="revoke-keys"
Jul 17 20:54:00.124805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124785    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b97cff-db42-4920-838c-39efd272e517" containerName="revoke-keys"
Jul 17 20:54:00.124805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124792    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:54:00.124805 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124797    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf5822d-8073-4cb6-b72e-a302f97c43d1" containerName="revoke-keys"
Jul 17 20:54:00.125064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124841    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b97cff-db42-4920-838c-39efd272e517" containerName="revoke-keys"
Jul 17 20:54:00.125064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124850    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b97cff-db42-4920-838c-39efd272e517" containerName="revoke-keys"
Jul 17 20:54:00.125064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124855    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b97cff-db42-4920-838c-39efd272e517" containerName="revoke-keys"
Jul 17 20:54:00.125064 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.124861    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="218141cf-1cc5-4e48-8e70-4e9bc0b4d857" containerName="revoke-keys"
Jul 17 20:54:00.128946 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.128927    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"
Jul 17 20:54:00.131260 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.131242    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kc768\""
Jul 17 20:54:00.131349 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.131281    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 20:54:00.134822 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.134787    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"]
Jul 17 20:54:00.243082 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.243025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa59971-6452-478b-9b11-2fe2d1ced9be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr\" (UID: \"0aa59971-6452-478b-9b11-2fe2d1ced9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"
Jul 17 20:54:00.344311 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.344282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa59971-6452-478b-9b11-2fe2d1ced9be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr\" (UID: \"0aa59971-6452-478b-9b11-2fe2d1ced9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"
Jul 17 20:54:00.344933 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.344907    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa59971-6452-478b-9b11-2fe2d1ced9be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr\" (UID: \"0aa59971-6452-478b-9b11-2fe2d1ced9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"
Jul 17 20:54:00.440485 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.440403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"
Jul 17 20:54:00.572977 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:00.572956    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"]
Jul 17 20:54:00.574887 ip-10-0-143-185 kubenswrapper[2572]: W0717 20:54:00.574857    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa59971_6452_478b_9b11_2fe2d1ced9be.slice/crio-adb1cca2548a14fc7c13b56b064f7541446a1a7eacbf0032213420abe32ff2b3 WatchSource:0}: Error finding container adb1cca2548a14fc7c13b56b064f7541446a1a7eacbf0032213420abe32ff2b3: Status 404 returned error can't find the container with id adb1cca2548a14fc7c13b56b064f7541446a1a7eacbf0032213420abe32ff2b3
Jul 17 20:54:01.089969 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:01.089889    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mfnl8_dcfa7f00-ae9c-4e06-a14d-937beeaa6381/discovery/0.log"
Jul 17 20:54:01.112653 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:01.112633    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fdbd865b7-2zgvv_be6d4ef4-252f-4dd5-9570-3eacd013e3b3/kube-auth-proxy/0.log"
Jul 17 20:54:01.263134 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:01.263097    2572 generic.go:356] "Generic (PLEG): container finished" podID="0aa59971-6452-478b-9b11-2fe2d1ced9be" containerID="3bf7a18103152a7b2d076bd0ace5b809c92e7bc8868eca9fd62303b49a66945f" exitCode=6
Jul 17 20:54:01.263320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:01.263160    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" event={"ID":"0aa59971-6452-478b-9b11-2fe2d1ced9be","Type":"ContainerDied","Data":"3bf7a18103152a7b2d076bd0ace5b809c92e7bc8868eca9fd62303b49a66945f"}
Jul 17 20:54:01.263320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:01.263228    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" event={"ID":"0aa59971-6452-478b-9b11-2fe2d1ced9be","Type":"ContainerStarted","Data":"adb1cca2548a14fc7c13b56b064f7541446a1a7eacbf0032213420abe32ff2b3"}
Jul 17 20:54:01.263320 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:01.263310    2572 scope.go:117] "RemoveContainer" containerID="3bf7a18103152a7b2d076bd0ace5b809c92e7bc8868eca9fd62303b49a66945f"
Jul 17 20:54:02.239497 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.239445    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pdldf_0e8996f1-c95e-4e7b-9f2e-e8bd0b08ddd0/serve-healthcheck-canary/0.log"
Jul 17 20:54:02.270185 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.270153    2572 generic.go:356] "Generic (PLEG): container finished" podID="0aa59971-6452-478b-9b11-2fe2d1ced9be" containerID="ecea614dc527643a87bcd0e081b16ec547f5a68dfc13996d8b277ee3cc8d05ad" exitCode=6
Jul 17 20:54:02.270308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.270238    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" event={"ID":"0aa59971-6452-478b-9b11-2fe2d1ced9be","Type":"ContainerDied","Data":"ecea614dc527643a87bcd0e081b16ec547f5a68dfc13996d8b277ee3cc8d05ad"}
Jul 17 20:54:02.270308 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.270298    2572 scope.go:117] "RemoveContainer" containerID="3bf7a18103152a7b2d076bd0ace5b809c92e7bc8868eca9fd62303b49a66945f"
Jul 17 20:54:02.270569 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.270553    2572 scope.go:117] "RemoveContainer" containerID="ecea614dc527643a87bcd0e081b16ec547f5a68dfc13996d8b277ee3cc8d05ad"
Jul 17 20:54:02.270793 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:54:02.270772    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr_odh-ai-gateway-infra(0aa59971-6452-478b-9b11-2fe2d1ced9be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" podUID="0aa59971-6452-478b-9b11-2fe2d1ced9be"
Jul 17 20:54:02.909435 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.909408    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvcr7_f98e6187-4523-4d1b-862c-18bc65445b93/kube-rbac-proxy/0.log"
Jul 17 20:54:02.933849 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.933822    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvcr7_f98e6187-4523-4d1b-862c-18bc65445b93/exporter/0.log"
Jul 17 20:54:02.966024 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:02.966001    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvcr7_f98e6187-4523-4d1b-862c-18bc65445b93/extractor/0.log"
Jul 17 20:54:03.276608 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:03.276585    2572 scope.go:117] "RemoveContainer" containerID="ecea614dc527643a87bcd0e081b16ec547f5a68dfc13996d8b277ee3cc8d05ad"
Jul 17 20:54:03.277015 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:54:03.276758    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr_odh-ai-gateway-infra(0aa59971-6452-478b-9b11-2fe2d1ced9be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" podUID="0aa59971-6452-478b-9b11-2fe2d1ced9be"
Jul 17 20:54:04.256933 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:04.256907    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wlh9g/perf-node-gather-daemonset-ff6h7"
Jul 17 20:54:05.261175 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:05.261146    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s5fvl_5f5c686c-fea2-4dd0-98e5-2f2549bbdfb9/manager/0.log"
Jul 17 20:54:05.404839 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:05.404792    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bxc4f_e9666413-1a40-4ad3-b2bc-113de3eb747f/manager/0.log"
Jul 17 20:54:05.528527 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:05.528421    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nhgzn_8e526790-634c-4370-8785-fa57241af7e9/server/0.log"
Jul 17 20:54:05.592598 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:05.592548    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mfjbf_ad0bb1e5-caf8-40cb-a9b8-7013d746e028/manager/0.log"
Jul 17 20:54:05.659397 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:05.659371    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm_74343d95-b829-4a9d-a2c5-35b7dc48c6b4/manager/0.log"
Jul 17 20:54:05.676934 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:05.676913    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bdc7ddc4-8c2hm_74343d95-b829-4a9d-a2c5-35b7dc48c6b4/copy-manifests/0.log"
Jul 17 20:54:07.364886 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:07.364854    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mfnl8_dcfa7f00-ae9c-4e06-a14d-937beeaa6381/discovery/0.log"
Jul 17 20:54:07.375604 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:07.375580    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fdbd865b7-2zgvv_be6d4ef4-252f-4dd5-9570-3eacd013e3b3/kube-auth-proxy/0.log"
Jul 17 20:54:08.935323 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:08.935292    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mfnl8_dcfa7f00-ae9c-4e06-a14d-937beeaa6381/discovery/0.log"
Jul 17 20:54:08.949605 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:08.949574    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fdbd865b7-2zgvv_be6d4ef4-252f-4dd5-9570-3eacd013e3b3/kube-auth-proxy/0.log"
Jul 17 20:54:13.847863 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:13.847833    2572 scope.go:117] "RemoveContainer" containerID="ecea614dc527643a87bcd0e081b16ec547f5a68dfc13996d8b277ee3cc8d05ad"
Jul 17 20:54:14.045908 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.045882    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jt22m_c048b030-3ada-4681-b3d7-3f177fa5122c/kube-multus-additional-cni-plugins/0.log"
Jul 17 20:54:14.069141 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.069114    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jt22m_c048b030-3ada-4681-b3d7-3f177fa5122c/egress-router-binary-copy/0.log"
Jul 17 20:54:14.091755 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.091727    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jt22m_c048b030-3ada-4681-b3d7-3f177fa5122c/cni-plugins/0.log"
Jul 17 20:54:14.115451 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.115393    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jt22m_c048b030-3ada-4681-b3d7-3f177fa5122c/bond-cni-plugin/0.log"
Jul 17 20:54:14.137086 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.137067    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jt22m_c048b030-3ada-4681-b3d7-3f177fa5122c/routeoverride-cni/0.log"
Jul 17 20:54:14.158666 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.158648    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jt22m_c048b030-3ada-4681-b3d7-3f177fa5122c/whereabouts-cni-bincopy/0.log"
Jul 17 20:54:14.182796 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.182775    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jt22m_c048b030-3ada-4681-b3d7-3f177fa5122c/whereabouts-cni/0.log"
Jul 17 20:54:14.329806 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.329774    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" event={"ID":"0aa59971-6452-478b-9b11-2fe2d1ced9be","Type":"ContainerStarted","Data":"c1bbe20fb0ffecfa7e619f78e54925918eb570bb821c8e3213db3207af093da4"}
Jul 17 20:54:14.347503 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.347451    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" podStartSLOduration=14.347435731000001 podStartE2EDuration="14.347435731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 20:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 20:54:14.345526143 +0000 UTC m=+3489.039502290" watchObservedRunningTime="2026-07-17 20:54:14.347435731 +0000 UTC m=+3489.041411878"
Jul 17 20:54:14.470073 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.470034    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vjss5_f6bfc390-7582-4a98-b9f7-0b89c792034c/kube-multus/0.log"
Jul 17 20:54:14.536761 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:54:14.536734    2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa59971_6452_478b_9b11_2fe2d1ced9be.slice/crio-c1bbe20fb0ffecfa7e619f78e54925918eb570bb821c8e3213db3207af093da4.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 20:54:14.544655 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.544622    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b5zcw_bf6b5042-e40a-4549-a761-2ef68ad8c71b/network-metrics-daemon/0.log"
Jul 17 20:54:14.561437 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:14.561414    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b5zcw_bf6b5042-e40a-4549-a761-2ef68ad8c71b/kube-rbac-proxy/0.log"
Jul 17 20:54:15.335797 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:15.335759    2572 generic.go:356] "Generic (PLEG): container finished" podID="0aa59971-6452-478b-9b11-2fe2d1ced9be" containerID="c1bbe20fb0ffecfa7e619f78e54925918eb570bb821c8e3213db3207af093da4" exitCode=6
Jul 17 20:54:15.336237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:15.335831    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" event={"ID":"0aa59971-6452-478b-9b11-2fe2d1ced9be","Type":"ContainerDied","Data":"c1bbe20fb0ffecfa7e619f78e54925918eb570bb821c8e3213db3207af093da4"}
Jul 17 20:54:15.336237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:15.335888    2572 scope.go:117] "RemoveContainer" containerID="ecea614dc527643a87bcd0e081b16ec547f5a68dfc13996d8b277ee3cc8d05ad"
Jul 17 20:54:15.336237 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:15.336151    2572 scope.go:117] "RemoveContainer" containerID="c1bbe20fb0ffecfa7e619f78e54925918eb570bb821c8e3213db3207af093da4"
Jul 17 20:54:15.336425 ip-10-0-143-185 kubenswrapper[2572]: E0717 20:54:15.336361    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr_odh-ai-gateway-infra(0aa59971-6452-478b-9b11-2fe2d1ced9be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr" podUID="0aa59971-6452-478b-9b11-2fe2d1ced9be"
Jul 17 20:54:15.363061 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:15.363027    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2e9331f5-xvblr"]
Jul 17 20:54:15.976495 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:15.976464    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9tgrl_c6fe3fbc-3b7d-4df5-9a09-12d85016fa68/ovn-controller/0.log"
Jul 17 20:54:15.994352 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:15.994316    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9tgrl_c6fe3fbc-3b7d-4df5-9a09-12d85016fa68/ovn-acl-logging/0.log"
Jul 17 20:54:16.018997 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:16.018971    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9tgrl_c6fe3fbc-3b7d-4df5-9a09-12d85016fa68/kube-rbac-proxy-node/0.log"
Jul 17 20:54:16.041519 ip-10-0-143-185 kubenswrapper[2572]: I0717 20:54:16.041498    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9tgrl_c6fe3fbc-3b7d-4df5-9a09-12d85016fa68/kube-rbac-proxy-ovn-metrics/0.log"