Jul 17 13:45:20.267853 ip-10-0-129-191 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.631329    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634414    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634425    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634428    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634431    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634434    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634436    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634439    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 13:45:20.670628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634441    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634444    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634447    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634449    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634451    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634455    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634458    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634460    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634463    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634466    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634469    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634480    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634483    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634486    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634488    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634491    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634493    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634496    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634499    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634501    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 13:45:20.672976 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634504    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634507    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634509    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634513    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634517    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634520    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634524    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634527    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634530    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634533    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634535    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634538    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634541    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634544    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634546    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634549    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634552    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634554    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 13:45:20.675344 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634557    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634559    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634564    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634567    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634570    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634573    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634575    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634577    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634580    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634582    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634585    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634588    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634590    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634593    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634596    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634598    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634601    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634603    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634606    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634624    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 13:45:20.677115 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634627    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634630    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634632    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634635    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634638    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634641    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634645    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634648    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634650    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634653    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634655    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634658    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634661    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634663    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634665    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634668    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634670    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634673    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634675    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634678    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 13:45:20.690724 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634681    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634683    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634686    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634689    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634691    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634694    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634696    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634699    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634702    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634704    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634707    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634709    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634712    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.634714    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635367    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635372    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635376    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635379    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635382    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635385    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 13:45:20.692628 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635388    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635391    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635394    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635396    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635399    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635401    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635404    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635406    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635409    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635412    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635414    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635416    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635419    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635422    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635425    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635427    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635429    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635432    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635435    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635437    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635439    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 13:45:20.695029 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635442    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635445    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635448    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635450    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635453    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635455    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635458    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635460    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635463    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635465    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635468    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635471    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635474    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635476    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635479    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635482    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635484    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635488    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635490    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635493    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 13:45:20.710385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635495    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 13:45:20.698522 ip-10-0-129-191 systemd[1]: Started Kubernetes Kubelet.
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635498    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635500    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635502    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635505    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635507    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635509    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635515    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635517    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635520    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635523    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635525    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635528    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635530    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635533    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635535    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635537    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635540    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635542    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635545    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 13:45:20.726716 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635548    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635550    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635552    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635555    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635558    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635561    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635563    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635566    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635568    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635571    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635574    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635576    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635578    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635581    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635584    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635587    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635590    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635592    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635595    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 13:45:20.729006 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635601    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635604    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635606    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635626    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635629    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635632    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635635    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635637    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635641    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635644    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635648    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635652    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635655    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.635997    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636001    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636004    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636006    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636010    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 13:45:20.743913 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636012    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636015    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636017    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636020    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636023    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636025    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636028    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636030    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636033    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636035    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636038    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636040    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636043    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636045    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636049    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636052    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636055    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636057    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636060    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636062    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 13:45:20.758359 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636065    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636067    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636070    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636072    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636075    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636077    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636079    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636084    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636086    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636089    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636091    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636093    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636097    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636099    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636102    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636104    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636107    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636109    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636111    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636114    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 13:45:20.761247 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636116    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636119    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636121    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636123    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636126    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636128    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636132    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636135    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636137    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636140    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636144    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636147    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636150    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636152    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636155    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636157    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636160    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636162    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636164    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 13:45:20.915456 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636168    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636170    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636173    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636175    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636177    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636180    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636183    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636186    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636188    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636190    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636193    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636195    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636197    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636200    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636202    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636205    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636207    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636211    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636215    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 13:45:20.917268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636220    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636224    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636227    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636230    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636232    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636235    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636238    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636240    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636243    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636245    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636248    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636250    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636252    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636257    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636259    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636261    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636327    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636340    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636346    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636351    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636355    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 13:45:20.917888 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636358    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636362    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636366    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636369    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636372    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636375    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636378    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636381    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636384    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636386    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636389    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636393    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636396    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636400    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636403    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636406    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636409    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636412    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636416    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636418    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636421    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636424    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636427    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636430    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 13:45:20.918558 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636435    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636438    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636440    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636444    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636447    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636450    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636453    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636456    2575 flags.go:64] FLAG: --enable-server="true"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636460    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636464    2575 flags.go:64] FLAG: --event-burst="100"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636467    2575 flags.go:64] FLAG: --event-qps="50"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636470    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636473    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636475    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636479    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636482    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636485    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636487    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636490    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636494    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636497    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636500    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636502    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636505    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636508    2575 flags.go:64] FLAG: --feature-gates=""
Jul 17 13:45:20.919250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636514    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636517    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636520    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636522    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636525    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636528    2575 flags.go:64] FLAG: --help="false"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636531    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-129-191.ec2.internal"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636535    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636538    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636540    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636543    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636547    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636549    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636552    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636555    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636557    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636561    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636564    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636567    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636570    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636572    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636575    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636578    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636581    2575 flags.go:64] FLAG: --lock-file=""
Jul 17 13:45:20.920142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636584    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636587    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636591    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636595    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636598    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636601    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636604    2575 flags.go:64] FLAG: --logging-format="text"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636623    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636627    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636629    2575 flags.go:64] FLAG: --manifest-url=""
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636632    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636636    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636639    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636643    2575 flags.go:64] FLAG: --max-pods="110"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636646    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636650    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636653    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636656    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636659    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636662    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636665    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636671    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636674    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636677    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 13:45:20.922755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636679    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636683    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636688    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636691    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636694    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636697    2575 flags.go:64] FLAG: --port="10250"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636700    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636702    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ee932e574c0a960a"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636705    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636709    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636711    2575 flags.go:64] FLAG: --register-node="true"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636716    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636719    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636722    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636725    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636727    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636731    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636734    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636736    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636739    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636742    2575 flags.go:64] FLAG: --runonce="false"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636744    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636747    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636751    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636754    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 13:45:20.923547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636757    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636760    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636763    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636765    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636768    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636771    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636774    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636777    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636780    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636783    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636785    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636790    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636793    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636796    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636800    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636802    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636805    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636808    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636812    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636815    2575 flags.go:64] FLAG: --v="2"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636819    2575 flags.go:64] FLAG: --version="false"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636823    2575 flags.go:64] FLAG: --vmodule=""
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636826    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.636829    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636934    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 13:45:20.930073 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636937    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636940    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636943    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636945    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636948    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636952    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636954    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636956    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636959    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636962    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636965    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636967    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636970    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636973    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636975    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636978    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636980    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636983    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636986    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636989    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 13:45:20.930902 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636991    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636994    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636996    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.636999    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637001    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637005    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637007    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637010    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637012    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637014    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637017    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637019    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637029    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637032    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637035    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637038    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637040    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637044    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637047    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 13:45:20.931540 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637049    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637052    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637056    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637059    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637062    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637065    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637068    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637071    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637073    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637076    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637078    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637082    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637084    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637087    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637089    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637091    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637094    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637097    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637100    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637103    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 13:45:20.932087 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637105    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637108    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637110    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637112    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637115    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637117    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637120    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637122    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637125    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637128    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637132    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637134    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637137    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637139    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637142    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637146    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637149    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637151    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637154    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 13:45:20.932593 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637156    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637159    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637161    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637163    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637166    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637173    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637175    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637179    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637182    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637185    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637188    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637192    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637195    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637198    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637201    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637203    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637206    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637208    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637211    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 13:45:20.933287 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.637213    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.637688    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.644806    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.644819    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.645767    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.645774    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645829    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645835    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645839    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645843    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645846    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645849    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645852    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645854    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 13:45:20.933925 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645857    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645860    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645862    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645865    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645867    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645870    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645872    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645875    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645877    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645879    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645882    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645886    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645888    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645891    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645893    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645896    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645898    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645901    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645909    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645912    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 13:45:20.934352 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645914    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645916    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645921    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645924    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645926    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645929    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645931    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645934    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645936    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645939    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645941    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645944    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645947    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645949    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645951    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645954    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645956    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645959    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645963    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645965    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 13:45:20.934991 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645967    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645970    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645972    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645976    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645979    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645981    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645983    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645986    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645989    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645991    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645995    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.645998    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646001    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646004    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646010    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646012    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646015    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646017    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646020    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 13:45:20.935695 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646022    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646025    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646027    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646030    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646032    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646034    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646037    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646039    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646042    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646044    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646047    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646049    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646052    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646054    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646057    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646060    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646064    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646066    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646069    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646071    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 13:45:20.936256 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646074    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646076    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646078    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646081    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646083    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646086    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646090    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646094    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646097    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646099    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646102    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646104    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.646109    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646220    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646225    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 13:45:21.017383 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646227    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646230    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646232    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646235    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646237    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646240    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646243    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646245    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646247    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646250    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646253    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646256    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646258    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646261    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646264    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646267    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646270    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646272    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646275    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646277    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 13:45:21.017942 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646279    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646282    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646285    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646288    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646293    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646296    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646299    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646302    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646305    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646308    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646310    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646313    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646316    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646318    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646320    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646323    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646327    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646330    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646332    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 13:45:21.018969 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646334    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646337    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646339    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646342    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646345    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646348    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646350    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646354    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646357    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646359    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646361    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646364    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646366    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646369    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646372    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646374    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646377    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646380    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646384    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646387    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 13:45:21.019594 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646390    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646393    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646396    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646399    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646401    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646404    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646406    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646409    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646411    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646413    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646416    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646418    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646421    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646423    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646426    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646428    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646431    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646434    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646436    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646440    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 13:45:21.020216 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646442    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646444    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646447    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646449    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646452    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646454    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646457    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646459    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646461    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646466    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646468    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646471    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646473    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646475    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646478    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646480    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646483    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 13:45:21.020726 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:20.646485    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.646490    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.646601    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.650196    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.651232    2575 server.go:1019] "Starting client certificate rotation"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.651329    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.651367    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.654113    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.657261    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.668367    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.670659    2575 log.go:25] "Validated CRI v1 image API"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.672125    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 13:45:21.021201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.674902    2575 fs.go:135] Filesystem UUIDs: map[5b672f87-f652-4b82-8f3d-c0e1356cf7e4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e3e69875-14d7-4e5a-b5f7-89a6037eb8bb:/dev/nvme0n1p4]
Jul 17 13:45:21.021534 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.674918    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.679700    2575 manager.go:216] Machine: {Timestamp:2026-07-17 13:45:20.678649839 +0000 UTC m=+0.293663256 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3205145 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22e505b2cd110629c2ad56d219c714 SystemUUID:ec22e505-b2cd-1106-29c2-ad56d219c714 BootID:a78ddf69-8750-4747-93d7-5ec863e9d0c6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b0:22:dd:1e:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b0:22:dd:1e:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:53:a1:b2:b6:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.679801    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.679869    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.682237    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.682258    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.682806    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.682816    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 13:45:21.021690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.682823    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.683508    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.684659    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.686987    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.687605    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.687719    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.687744    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.687753    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.687762    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.687771    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.687779    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.688046    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.688058    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.691969    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693315    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693341    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693915    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693934    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693940    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693945    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693954    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693959    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693967    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693972    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693978    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693983    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693991    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.693996    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.694601    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.694633    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.697746    2575 server.go:1268] "Started kubelet"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.697770    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.697949    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.698003    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.698519    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.698560    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.698703    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.698924    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.698946    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 13:45:21.022356 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.703119    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.702817    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-191.ec2.internal.18c317afaf5d37ea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-191.ec2.internal,UID:ip-10-0-129-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-191.ec2.internal,},FirstTimestamp:2026-07-17 13:45:20.697685994 +0000 UTC m=+0.312699411,LastTimestamp:2026-07-17 13:45:20.697685994 +0000 UTC m=+0.312699411,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-191.ec2.internal,}"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.706430    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.706447    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.706521    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707064    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707070    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707092    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707173    2575 factory.go:55] Registering systemd factory
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707218    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707224    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707233    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707385    2575 factory.go:153] Registering CRI-O factory
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707397    2575 factory.go:223] Registration of the crio container factory successfully
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707439    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707461    2575 factory.go:103] Registering Raw factory
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707478    2575 manager.go:1196] Started watching for new ooms in manager
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.707851    2575 manager.go:349] Starting recovery of all containers
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.708508    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.708792    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.718125    2575 manager.go:354] Recovery completed
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.721887    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.723509    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.723532    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.723541    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.723989    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.723997    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.724002    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.724018    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.724554    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.724605    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.724635    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.725110    2575 policy_none.go:49] "None policy: Start"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.725124    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 13:45:21.023386 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.725136    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.726099    2575 policy_none.go:47] "Start"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.726115    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5lz9"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.733334    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-191.ec2.internal.18c317afb0e76b1f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-191.ec2.internal,UID:ip-10-0-129-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-191.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-191.ec2.internal,},FirstTimestamp:2026-07-17 13:45:20.723520287 +0000 UTC m=+0.338533703,LastTimestamp:2026-07-17 13:45:20.723520287 +0000 UTC m=+0.338533703,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-191.ec2.internal,}"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.734331    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5lz9"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.738259    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-191.ec2.internal.18c317afb0e7a899  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-191.ec2.internal,UID:ip-10-0-129-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-191.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-191.ec2.internal,},FirstTimestamp:2026-07-17 13:45:20.723536025 +0000 UTC m=+0.338549442,LastTimestamp:2026-07-17 13:45:20.723536025 +0000 UTC m=+0.338549442,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-191.ec2.internal,}"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.790508    2575 manager.go:337] "Starting Device Plugin manager"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.790553    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.790564    2575 server.go:87] "Starting device plugin registration server"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.790792    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.790803    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.790983    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.791047    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.791054    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.791492    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.791543    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.839554    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.840591    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.840632    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.840653    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.840682    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.843333    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.891511    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.892333    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.892360    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.892374    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:21.112374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.892396    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.900629    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.900647    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-191.ec2.internal\": node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.915845    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.941451    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal"]
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.941501    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.942224    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.942249    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.942260    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.945365    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.945494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.945522    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.945998    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.946020    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.946029    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.946082    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.946101    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.946113    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.948113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.948576    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.949309    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.949342    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:20.949351    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:20.995123    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-191.ec2.internal\" not found" node="ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:21.000156    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-191.ec2.internal\" not found" node="ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.009389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f4b672f95d6ba53b17d4bc79ab9528-config\") pod \"kube-apiserver-proxy-ip-10-0-129-191.ec2.internal\" (UID: \"40f4b672f95d6ba53b17d4bc79ab9528\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.009412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/606cd06d04278e78e7756c2d28e16f16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal\" (UID: \"606cd06d04278e78e7756c2d28e16f16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.113217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.009428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/606cd06d04278e78e7756c2d28e16f16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal\" (UID: \"606cd06d04278e78e7756c2d28e16f16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.114058 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:21.016454    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:21.114058 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.109648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/606cd06d04278e78e7756c2d28e16f16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal\" (UID: \"606cd06d04278e78e7756c2d28e16f16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.114058 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.109671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/606cd06d04278e78e7756c2d28e16f16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal\" (UID: \"606cd06d04278e78e7756c2d28e16f16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.114058 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.109686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f4b672f95d6ba53b17d4bc79ab9528-config\") pod \"kube-apiserver-proxy-ip-10-0-129-191.ec2.internal\" (UID: \"40f4b672f95d6ba53b17d4bc79ab9528\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.114058 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.109714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f4b672f95d6ba53b17d4bc79ab9528-config\") pod \"kube-apiserver-proxy-ip-10-0-129-191.ec2.internal\" (UID: \"40f4b672f95d6ba53b17d4bc79ab9528\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.114058 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.109737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/606cd06d04278e78e7756c2d28e16f16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal\" (UID: \"606cd06d04278e78e7756c2d28e16f16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.114058 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.109749    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/606cd06d04278e78e7756c2d28e16f16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal\" (UID: \"606cd06d04278e78e7756c2d28e16f16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.212588 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:21.117412    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:21.302942 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:21.218342    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:21.302942 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.296522    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.303589 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:21.303160    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:21.409941 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:21.318662    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:22.740614 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:21.419218    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:22.747944 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:22.747378    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:22.915813 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:22.847858    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:22.915813 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:22.878998    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 13:45:22.915813 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:22.879112    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 13:45:22.986293 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:22.948779    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:22.986293 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:22.966160    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 13:40:21 +0000 UTC" deadline="2027-12-23 05:08:22.23926902 +0000 UTC"
Jul 17 13:45:22.986293 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:22.966182    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12567h22m59.273089861s"
Jul 17 13:45:23.070167 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.006117    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 13:45:23.070167 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:23.049702    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.121897    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.122983    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.123007    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.123016    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeHasSufficientPID"
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:23.135205    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f4b672f95d6ba53b17d4bc79ab9528.slice/crio-847bb3ed51df6a98c0ac64e2ebe537d3c1cdb70f2ffaa4eed428ad0412fe09d0 WatchSource:0}: Error finding container 847bb3ed51df6a98c0ac64e2ebe537d3c1cdb70f2ffaa4eed428ad0412fe09d0: Status 404 returned error can't find the container with id 847bb3ed51df6a98c0ac64e2ebe537d3c1cdb70f2ffaa4eed428ad0412fe09d0
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:23.135661    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606cd06d04278e78e7756c2d28e16f16.slice/crio-58c298437b0d292a86df9da5b5138eba1038fd7ea137b63af960ea2679e455f3 WatchSource:0}: Error finding container 58c298437b0d292a86df9da5b5138eba1038fd7ea137b63af960ea2679e455f3: Status 404 returned error can't find the container with id 58c298437b0d292a86df9da5b5138eba1038fd7ea137b63af960ea2679e455f3
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.141271    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:45:23.178740 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:23.150372    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:23.282892 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.207466    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 13:45:23.282892 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:23.250731    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-191.ec2.internal\" not found"
Jul 17 13:45:23.367552 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.331442    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 13:45:23.367552 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.334293    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:23.367552 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.347886    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 13:45:23.367552 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.348727    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal"
Jul 17 13:45:23.367552 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.359725    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 13:45:23.916383 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.916354    2575 apiserver.go:52] "Watching apiserver"
Jul 17 13:45:23.924798 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.924606    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 13:45:23.925060 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.925034    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-cr2qc","openshift-network-diagnostics/network-check-target-cpj75","openshift-network-operator/iptables-alerter-wb66s","kube-system/konnectivity-agent-t47gn","openshift-cluster-node-tuning-operator/tuned-q4xsq","openshift-dns/node-resolver-58ggv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-88s54","openshift-multus/multus-tj2bb","openshift-ovn-kubernetes/ovnkube-node-fhqvc","kube-system/global-pull-secret-syncer-hq7zk","kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb","openshift-image-registry/node-ca-gvl6k"]
Jul 17 13:45:23.926752 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.926731    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.929889 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.928887    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.929889 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.929475    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 13:45:23.929889 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.929668    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 13:45:23.929889 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.929879    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvrtp\""
Jul 17 13:45:23.930179 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.930164    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:23.930179 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.930172    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 13:45:23.930616 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.930466    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 13:45:23.930616 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.930479    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 13:45:23.930772 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.930711    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 13:45:23.931143 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.931118    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wjsrq\""
Jul 17 13:45:23.931370 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.931355    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 13:45:23.932065 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.931612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:23.932065 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.932027    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 13:45:23.932549 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.932424    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 13:45:23.933349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.932845    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 13:45:23.933349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.932862    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 13:45:23.933349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.933036    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 13:45:23.933349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.933098    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9l7m4\""
Jul 17 13:45:23.933349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.933307    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.934686 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.934042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:23.934864 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.934847    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mkr9q\""
Jul 17 13:45:23.935485 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.935085    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 13:45:23.935485 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.935185    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 13:45:23.935485 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.935363    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 13:45:23.936448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.935835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.936448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.936123    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 13:45:23.936448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.936132    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 13:45:23.936448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.936324    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 13:45:23.936822 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.936597    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cvf66\""
Jul 17 13:45:23.936822 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.936714    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kvc82\""
Jul 17 13:45:23.936822 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.936738    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 13:45:23.937308 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.937049    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:23.938548 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.938506    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:23.940860 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:23.939272    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:23.942177 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.942159    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 13:45:23.942512 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.942493    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 13:45:23.942856 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.942831    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 13:45:23.942979 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.942653    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 13:45:23.943044 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.942960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:23.943090 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:23.943059    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:23.943090 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.942563    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 13:45:23.943193 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.943092    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xd7qw\""
Jul 17 13:45:23.943238 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.943195    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qqgc7\""
Jul 17 13:45:23.943617 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.943602    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 13:45:23.944255 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.944149    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 13:45:23.945204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d482fb-3510-4df0-8302-71ca87962c0b-tmp\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.945204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-hosts-file\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:23.945204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945108    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-os-release\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.945204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-k8s-cni-cncf-io\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.945204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:23.945204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xg9\" (UniqueName: \"kubernetes.io/projected/61c8b7ed-7cda-4535-9499-9bfe558d39c0-kube-api-access-z8xg9\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.945204 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:23.945199    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-slash\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-cni-netd\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-registration-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3405f65-f7c9-4913-8d29-611a74961d64-konnectivity-ca\") pod \"konnectivity-agent-t47gn\" (UID: \"b3405f65-f7c9-4913-8d29-611a74961d64\") " pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39d27342-9a1d-4bc9-befe-1f2c30b84406-iptables-alerter-script\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-sys\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61c8b7ed-7cda-4535-9499-9bfe558d39c0-cni-binary-copy\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-multus-certs\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-kubelet\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945521    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.945670 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945545    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-systemd\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39d27342-9a1d-4bc9-befe-1f2c30b84406-host-slash\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-socket-dir-parent\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945612    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-var-lib-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-env-overrides\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysctl-conf\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-var-lib-kubelet\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-host\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-node-log\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-device-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sl4\" (UniqueName: \"kubernetes.io/projected/39d27342-9a1d-4bc9-befe-1f2c30b84406-kube-api-access-84sl4\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthqx\" (UniqueName: \"kubernetes.io/projected/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-kube-api-access-wthqx\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-ovnkube-script-lib\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3405f65-f7c9-4913-8d29-611a74961d64-agent-certs\") pod \"konnectivity-agent-t47gn\" (UID: \"b3405f65-f7c9-4913-8d29-611a74961d64\") " pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:23.946204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.945991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysctl-d\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-modprobe-d\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946034    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysconfig\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-kubernetes\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-system-cni-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-daemon-config\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-run-netns\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-conf-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-systemd-units\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwz7\" (UniqueName: \"kubernetes.io/projected/85d482fb-3510-4df0-8302-71ca87962c0b-kube-api-access-gzwz7\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946199    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-tmp-dir\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946221    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-cni-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-kubelet\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-hostroot\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.946935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946329    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-ovn\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-run-ovn-kubernetes\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946469    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-netns\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-run\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946557    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-cni-bin\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-lib-modules\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85d482fb-3510-4df0-8302-71ca87962c0b-etc-tuned\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946697    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-etc-kubernetes\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-log-socket\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946741    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13dc9407-48bb-4162-b663-f642fb0a6248-ovn-node-metrics-cert\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946761    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-socket-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946783    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-sys-fs\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.947579 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvfp\" (UniqueName: \"kubernetes.io/projected/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-kube-api-access-cwvfp\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:23.948196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-cni-multus\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.948196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-etc-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.948196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946957    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-ovnkube-config\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.948196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.946986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-systemd\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:23.948196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.947029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-cnibin\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.948196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.947135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-cni-bin\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:23.948196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.947160    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m82c\" (UniqueName: \"kubernetes.io/projected/13dc9407-48bb-4162-b663-f642fb0a6248-kube-api-access-6m82c\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:23.949592 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.949564    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-94xm6\""
Jul 17 13:45:23.949592 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.949622    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 13:45:23.949848 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.949768    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 13:45:23.949895 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.949873    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 13:45:23.983364 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.983337    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vxjf5"
Jul 17 13:45:23.995163 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:23.995139    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vxjf5"
Jul 17 13:45:24.034998 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.034940    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 13:45:24.035893 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.035856    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 13:45:24.047328 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-var-lib-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.047419 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-env-overrides\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.047419 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-serviceca\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.047419 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-cnibin\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.047419 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047405    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-var-lib-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.047419 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysctl-conf\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-var-lib-kubelet\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-host\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-node-log\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-device-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sl4\" (UniqueName: \"kubernetes.io/projected/39d27342-9a1d-4bc9-befe-1f2c30b84406-kube-api-access-84sl4\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d5310a1-8175-4478-b1c4-28ca2399c86b-dbus\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.047664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthqx\" (UniqueName: \"kubernetes.io/projected/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-kube-api-access-wthqx\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-ovnkube-script-lib\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3405f65-f7c9-4913-8d29-611a74961d64-agent-certs\") pod \"konnectivity-agent-t47gn\" (UID: \"b3405f65-f7c9-4913-8d29-611a74961d64\") " pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d5310a1-8175-4478-b1c4-28ca2399c86b-kubelet-config\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysctl-d\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047834    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysctl-conf\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-modprobe-d\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysconfig\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-var-lib-kubelet\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-kubernetes\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047901    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-env-overrides\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-system-cni-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-host\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-daemon-config\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-run-netns\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.048047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047995    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-device-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.047998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-system-cni-dir\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnwx\" (UniqueName: \"kubernetes.io/projected/d8c704e5-f773-48ec-9d76-717630b9aecb-kube-api-access-lxnwx\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-conf-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-systemd-units\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-host\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysctl-d\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwz7\" (UniqueName: \"kubernetes.io/projected/85d482fb-3510-4df0-8302-71ca87962c0b-kube-api-access-gzwz7\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-tmp-dir\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-cni-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-kubelet\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-kubelet\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.048746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-node-log\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048361    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-ovnkube-script-lib\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-hostroot\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-hostroot\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-ovn\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-run-ovn-kubernetes\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-netns\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-run\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048542    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-cni-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048548    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-tmp-dir\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-cni-bin\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048605    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-run-netns\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-lib-modules\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.049261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85d482fb-3510-4df0-8302-71ca87962c0b-etc-tuned\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-modprobe-d\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048663    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-etc-kubernetes\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048672    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-log-socket\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13dc9407-48bb-4162-b663-f642fb0a6248-ovn-node-metrics-cert\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-system-cni-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048742    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-run\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-socket-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048784    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-ovn\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-sysconfig\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048850    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-lib-modules\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-sys-fs\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvfp\" (UniqueName: \"kubernetes.io/projected/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-kube-api-access-cwvfp\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-cni-multus\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.049922 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.048974    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-log-socket\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-etc-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-daemon-config\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-socket-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-ovnkube-config\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049098    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-run-ovn-kubernetes\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-os-release\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049131    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-etc-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049162    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-netns\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-systemd\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-cnibin\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049194    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-cni-multus\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-cni-bin\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.050627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-systemd\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m82c\" (UniqueName: \"kubernetes.io/projected/13dc9407-48bb-4162-b663-f642fb0a6248-kube-api-access-6m82c\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049272    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-cnibin\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-cni-bin\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzjz\" (UniqueName: \"kubernetes.io/projected/8ec62e35-b2e6-4161-871f-3463c827961d-kube-api-access-kqzjz\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-systemd-units\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-sys-fs\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-etc-kubernetes\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d482fb-3510-4df0-8302-71ca87962c0b-tmp\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-var-lib-cni-bin\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049395    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-hosts-file\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-os-release\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-hosts-file\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-k8s-cni-cncf-io\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xg9\" (UniqueName: \"kubernetes.io/projected/61c8b7ed-7cda-4535-9499-9bfe558d39c0-kube-api-access-z8xg9\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049496    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-etc-kubernetes\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-slash\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049531    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13dc9407-48bb-4162-b663-f642fb0a6248-ovnkube-config\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-cni-netd\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-k8s-cni-cncf-io\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049586    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-slash\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049623    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-os-release\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-registration-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049653    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-conf-dir\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049682    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-openvswitch\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-registration-dir\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.051927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3405f65-f7c9-4913-8d29-611a74961d64-konnectivity-ca\") pod \"konnectivity-agent-t47gn\" (UID: \"b3405f65-f7c9-4913-8d29-611a74961d64\") " pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049725    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-cni-netd\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39d27342-9a1d-4bc9-befe-1f2c30b84406-iptables-alerter-script\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twkn6\" (UniqueName: \"kubernetes.io/projected/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-kube-api-access-twkn6\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-cni-binary-copy\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.049966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-sys\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050013    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85d482fb-3510-4df0-8302-71ca87962c0b-sys\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61c8b7ed-7cda-4535-9499-9bfe558d39c0-cni-binary-copy\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-multus-certs\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-kubelet\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050113    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-host-run-multus-certs\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.052557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-systemd\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-host-kubelet\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39d27342-9a1d-4bc9-befe-1f2c30b84406-host-slash\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050309    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39d27342-9a1d-4bc9-befe-1f2c30b84406-iptables-alerter-script\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3405f65-f7c9-4913-8d29-611a74961d64-konnectivity-ca\") pod \"konnectivity-agent-t47gn\" (UID: \"b3405f65-f7c9-4913-8d29-611a74961d64\") " pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-socket-dir-parent\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050339    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13dc9407-48bb-4162-b663-f642fb0a6248-run-systemd\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050366    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39d27342-9a1d-4bc9-befe-1f2c30b84406-host-slash\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61c8b7ed-7cda-4535-9499-9bfe558d39c0-multus-socket-dir-parent\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.050551    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61c8b7ed-7cda-4535-9499-9bfe558d39c0-cni-binary-copy\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.052191    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d482fb-3510-4df0-8302-71ca87962c0b-tmp\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.052473    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3405f65-f7c9-4913-8d29-611a74961d64-agent-certs\") pod \"konnectivity-agent-t47gn\" (UID: \"b3405f65-f7c9-4913-8d29-611a74961d64\") " pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.052598    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85d482fb-3510-4df0-8302-71ca87962c0b-etc-tuned\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.053149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.052974    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13dc9407-48bb-4162-b663-f642fb0a6248-ovn-node-metrics-cert\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.058320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.058157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sl4\" (UniqueName: \"kubernetes.io/projected/39d27342-9a1d-4bc9-befe-1f2c30b84406-kube-api-access-84sl4\") pod \"iptables-alerter-wb66s\" (UID: \"39d27342-9a1d-4bc9-befe-1f2c30b84406\") " pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:24.058320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.058302    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwz7\" (UniqueName: \"kubernetes.io/projected/85d482fb-3510-4df0-8302-71ca87962c0b-kube-api-access-gzwz7\") pod \"tuned-q4xsq\" (UID: \"85d482fb-3510-4df0-8302-71ca87962c0b\") " pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.058320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.058412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvfp\" (UniqueName: \"kubernetes.io/projected/060fa2ff-277d-4d2a-9a45-533cc45fdb8a-kube-api-access-cwvfp\") pod \"aws-ebs-csi-driver-node-jj7qb\" (UID: \"060fa2ff-277d-4d2a-9a45-533cc45fdb8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.059380 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.059257    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthqx\" (UniqueName: \"kubernetes.io/projected/ef9b353f-aeb1-4d0b-adfb-01e472913c4b-kube-api-access-wthqx\") pod \"node-resolver-58ggv\" (UID: \"ef9b353f-aeb1-4d0b-adfb-01e472913c4b\") " pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:24.059821 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.059776    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xg9\" (UniqueName: \"kubernetes.io/projected/61c8b7ed-7cda-4535-9499-9bfe558d39c0-kube-api-access-z8xg9\") pod \"multus-tj2bb\" (UID: \"61c8b7ed-7cda-4535-9499-9bfe558d39c0\") " pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.060756 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.060480    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m82c\" (UniqueName: \"kubernetes.io/projected/13dc9407-48bb-4162-b663-f642fb0a6248-kube-api-access-6m82c\") pod \"ovnkube-node-fhqvc\" (UID: \"13dc9407-48bb-4162-b663-f642fb0a6248\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.076699 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.076640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal" event={"ID":"606cd06d04278e78e7756c2d28e16f16","Type":"ContainerStarted","Data":"58c298437b0d292a86df9da5b5138eba1038fd7ea137b63af960ea2679e455f3"}
Jul 17 13:45:24.077882 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.077853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal" event={"ID":"40f4b672f95d6ba53b17d4bc79ab9528","Type":"ContainerStarted","Data":"847bb3ed51df6a98c0ac64e2ebe537d3c1cdb70f2ffaa4eed428ad0412fe09d0"}
Jul 17 13:45:24.151623 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151593    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-os-release\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.151752 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.151752 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzjz\" (UniqueName: \"kubernetes.io/projected/8ec62e35-b2e6-4161-871f-3463c827961d-kube-api-access-kqzjz\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:24.151752 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkn6\" (UniqueName: \"kubernetes.io/projected/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-kube-api-access-twkn6\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.151752 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-os-release\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.151975 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.151975 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-cni-binary-copy\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.151975 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.151975 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.151905    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.151894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.151976    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.151991    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret podName:2d5310a1-8175-4478-b1c4-28ca2399c86b nodeName:}" failed. No retries permitted until 2026-07-17 13:45:24.651968736 +0000 UTC m=+3.039316169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret") pod "global-pull-secret-syncer-hq7zk" (UID: "2d5310a1-8175-4478-b1c4-28ca2399c86b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-serviceca\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.152060    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:45:24.652039331 +0000 UTC m=+3.039386767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-cnibin\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152156 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d5310a1-8175-4478-b1c4-28ca2399c86b-dbus\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d5310a1-8175-4478-b1c4-28ca2399c86b-kubelet-config\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-system-cni-dir\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnwx\" (UniqueName: \"kubernetes.io/projected/d8c704e5-f773-48ec-9d76-717630b9aecb-kube-api-access-lxnwx\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-host\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152465    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-cni-binary-copy\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-serviceca\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-cnibin\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152518    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8c704e5-f773-48ec-9d76-717630b9aecb-system-cni-dir\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-host\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.152561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d5310a1-8175-4478-b1c4-28ca2399c86b-dbus\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.152952 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152567    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d5310a1-8175-4478-b1c4-28ca2399c86b-kubelet-config\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.152952 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152647    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.152952 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.152888    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8c704e5-f773-48ec-9d76-717630b9aecb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.158562 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.158538    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 13:45:24.158654 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.158565    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 13:45:24.158654 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.158577    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4wrzc for pod openshift-network-diagnostics/network-check-target-cpj75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:24.158654 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.158637    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc podName:c0c4f750-ba8d-4262-8697-7369537dcd67 nodeName:}" failed. No retries permitted until 2026-07-17 13:45:24.658623846 +0000 UTC m=+3.045971264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4wrzc" (UniqueName: "kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc") pod "network-check-target-cpj75" (UID: "c0c4f750-ba8d-4262-8697-7369537dcd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:24.161114 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.161086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twkn6\" (UniqueName: \"kubernetes.io/projected/c74d1625-9ecb-45cd-bc27-3f87ed8d40b3-kube-api-access-twkn6\") pod \"node-ca-gvl6k\" (UID: \"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3\") " pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.161295 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.161259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzjz\" (UniqueName: \"kubernetes.io/projected/8ec62e35-b2e6-4161-871f-3463c827961d-kube-api-access-kqzjz\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:24.161941 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.161899    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnwx\" (UniqueName: \"kubernetes.io/projected/d8c704e5-f773-48ec-9d76-717630b9aecb-kube-api-access-lxnwx\") pod \"multus-additional-cni-plugins-88s54\" (UID: \"d8c704e5-f773-48ec-9d76-717630b9aecb\") " pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.242858 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.242773    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:24.249514 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.249489    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb"
Jul 17 13:45:24.261314 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.261291    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wb66s"
Jul 17 13:45:24.274352 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.274327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:24.282349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.282331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4xsq"
Jul 17 13:45:24.288880 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.288861    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58ggv"
Jul 17 13:45:24.296714 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.296692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tj2bb"
Jul 17 13:45:24.306132 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.306111    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gvl6k"
Jul 17 13:45:24.311659 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.311641    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-88s54"
Jul 17 13:45:24.656696 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.656618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:24.656696 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.656655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:24.656909 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.656786    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:24.656909 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.656842    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:45:25.656829158 +0000 UTC m=+4.044176589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:24.656909 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.656787    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:24.657028 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.656912    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret podName:2d5310a1-8175-4478-b1c4-28ca2399c86b nodeName:}" failed. No retries permitted until 2026-07-17 13:45:25.656899693 +0000 UTC m=+4.044247125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret") pod "global-pull-secret-syncer-hq7zk" (UID: "2d5310a1-8175-4478-b1c4-28ca2399c86b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:24.757988 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.757956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:24.758148 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.758129    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 13:45:24.758186 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.758156    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 13:45:24.758186 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.758166    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4wrzc for pod openshift-network-diagnostics/network-check-target-cpj75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:24.758297 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:24.758209    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc podName:c0c4f750-ba8d-4262-8697-7369537dcd67 nodeName:}" failed. No retries permitted until 2026-07-17 13:45:25.758196282 +0000 UTC m=+4.145543705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4wrzc" (UniqueName: "kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc") pod "network-check-target-cpj75" (UID: "c0c4f750-ba8d-4262-8697-7369537dcd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:24.908378 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.908350    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d27342_9a1d_4bc9_befe_1f2c30b84406.slice/crio-5b290e2f146e28c778b82385a498f37dabe330449580f019aad5332211ef74ad WatchSource:0}: Error finding container 5b290e2f146e28c778b82385a498f37dabe330449580f019aad5332211ef74ad: Status 404 returned error can't find the container with id 5b290e2f146e28c778b82385a498f37dabe330449580f019aad5332211ef74ad
Jul 17 13:45:24.926521 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.926489    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3405f65_f7c9_4913_8d29_611a74961d64.slice/crio-2feae6e54b72d2e498da9f27bfbd6a5a99f896e8fe8d03ecdc730f664c196006 WatchSource:0}: Error finding container 2feae6e54b72d2e498da9f27bfbd6a5a99f896e8fe8d03ecdc730f664c196006: Status 404 returned error can't find the container with id 2feae6e54b72d2e498da9f27bfbd6a5a99f896e8fe8d03ecdc730f664c196006
Jul 17 13:45:24.929348 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.929328    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dc9407_48bb_4162_b663_f642fb0a6248.slice/crio-8cde2eb5f35f12725144f2b4cf557f24d2e486b26bcd81c709656a3778c0759c WatchSource:0}: Error finding container 8cde2eb5f35f12725144f2b4cf557f24d2e486b26bcd81c709656a3778c0759c: Status 404 returned error can't find the container with id 8cde2eb5f35f12725144f2b4cf557f24d2e486b26bcd81c709656a3778c0759c
Jul 17 13:45:24.930362 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.930337    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74d1625_9ecb_45cd_bc27_3f87ed8d40b3.slice/crio-ecece5060b4d5353d6d769c0b55ed63dd284c5acaf75f916cc820018f688c924 WatchSource:0}: Error finding container ecece5060b4d5353d6d769c0b55ed63dd284c5acaf75f916cc820018f688c924: Status 404 returned error can't find the container with id ecece5060b4d5353d6d769c0b55ed63dd284c5acaf75f916cc820018f688c924
Jul 17 13:45:24.931101 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.930908    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c704e5_f773_48ec_9d76_717630b9aecb.slice/crio-9bd0737c5e48f4e54c0288cf0f5c398e9931745ac02d2b899661c3956fb59e81 WatchSource:0}: Error finding container 9bd0737c5e48f4e54c0288cf0f5c398e9931745ac02d2b899661c3956fb59e81: Status 404 returned error can't find the container with id 9bd0737c5e48f4e54c0288cf0f5c398e9931745ac02d2b899661c3956fb59e81
Jul 17 13:45:24.932300 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.932235    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060fa2ff_277d_4d2a_9a45_533cc45fdb8a.slice/crio-253712164b1c849759c073d37513167c58c7b4f1a00a9b894ed80fb2a7839055 WatchSource:0}: Error finding container 253712164b1c849759c073d37513167c58c7b4f1a00a9b894ed80fb2a7839055: Status 404 returned error can't find the container with id 253712164b1c849759c073d37513167c58c7b4f1a00a9b894ed80fb2a7839055
Jul 17 13:45:24.933268 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.933181    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9b353f_aeb1_4d0b_adfb_01e472913c4b.slice/crio-3d83a0952f441a75efa23677c349711edaa811bbb7bc46128e4d79c7b4beaad8 WatchSource:0}: Error finding container 3d83a0952f441a75efa23677c349711edaa811bbb7bc46128e4d79c7b4beaad8: Status 404 returned error can't find the container with id 3d83a0952f441a75efa23677c349711edaa811bbb7bc46128e4d79c7b4beaad8
Jul 17 13:45:24.934068 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.933780    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c8b7ed_7cda_4535_9499_9bfe558d39c0.slice/crio-5eafee2fd49f1cc160380f4e1123660f8cb2e8c922d4edcec374f07b40085dfc WatchSource:0}: Error finding container 5eafee2fd49f1cc160380f4e1123660f8cb2e8c922d4edcec374f07b40085dfc: Status 404 returned error can't find the container with id 5eafee2fd49f1cc160380f4e1123660f8cb2e8c922d4edcec374f07b40085dfc
Jul 17 13:45:24.934957 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:24.934936    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d482fb_3510_4df0_8302_71ca87962c0b.slice/crio-a2720d85eeb8f37cd6dcb5979807a284fb061681e71cb1e6d8dc067c77e3a4c0 WatchSource:0}: Error finding container a2720d85eeb8f37cd6dcb5979807a284fb061681e71cb1e6d8dc067c77e3a4c0: Status 404 returned error can't find the container with id a2720d85eeb8f37cd6dcb5979807a284fb061681e71cb1e6d8dc067c77e3a4c0
Jul 17 13:45:24.997824 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.997744    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 13:40:23 +0000 UTC" deadline="2027-12-22 19:21:27.089226257 +0000 UTC"
Jul 17 13:45:24.997824 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:24.997797    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12557h36m2.09143385s"
Jul 17 13:45:25.069213 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.069076    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:25.069361 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.069085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:25.069361 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.069331    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:25.069473 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.069383    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:25.080704 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.080672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal" event={"ID":"40f4b672f95d6ba53b17d4bc79ab9528","Type":"ContainerStarted","Data":"15bdb04c9838969038396e415580e5424537e95e6e7ce0e04edea654d97de611"}
Jul 17 13:45:25.081775 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.081743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gvl6k" event={"ID":"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3","Type":"ContainerStarted","Data":"ecece5060b4d5353d6d769c0b55ed63dd284c5acaf75f916cc820018f688c924"}
Jul 17 13:45:25.082692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.082672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58ggv" event={"ID":"ef9b353f-aeb1-4d0b-adfb-01e472913c4b","Type":"ContainerStarted","Data":"3d83a0952f441a75efa23677c349711edaa811bbb7bc46128e4d79c7b4beaad8"}
Jul 17 13:45:25.083575 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.083555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb" event={"ID":"060fa2ff-277d-4d2a-9a45-533cc45fdb8a","Type":"ContainerStarted","Data":"253712164b1c849759c073d37513167c58c7b4f1a00a9b894ed80fb2a7839055"}
Jul 17 13:45:25.084444 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.084426    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerStarted","Data":"9bd0737c5e48f4e54c0288cf0f5c398e9931745ac02d2b899661c3956fb59e81"}
Jul 17 13:45:25.085317 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.085296    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"8cde2eb5f35f12725144f2b4cf557f24d2e486b26bcd81c709656a3778c0759c"}
Jul 17 13:45:25.086312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.086267    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t47gn" event={"ID":"b3405f65-f7c9-4913-8d29-611a74961d64","Type":"ContainerStarted","Data":"2feae6e54b72d2e498da9f27bfbd6a5a99f896e8fe8d03ecdc730f664c196006"}
Jul 17 13:45:25.087185 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.087162    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tj2bb" event={"ID":"61c8b7ed-7cda-4535-9499-9bfe558d39c0","Type":"ContainerStarted","Data":"5eafee2fd49f1cc160380f4e1123660f8cb2e8c922d4edcec374f07b40085dfc"}
Jul 17 13:45:25.088024 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.088009    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4xsq" event={"ID":"85d482fb-3510-4df0-8302-71ca87962c0b","Type":"ContainerStarted","Data":"a2720d85eeb8f37cd6dcb5979807a284fb061681e71cb1e6d8dc067c77e3a4c0"}
Jul 17 13:45:25.088872 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.088854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wb66s" event={"ID":"39d27342-9a1d-4bc9-befe-1f2c30b84406","Type":"ContainerStarted","Data":"5b290e2f146e28c778b82385a498f37dabe330449580f019aad5332211ef74ad"}
Jul 17 13:45:25.093906 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.093864    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-191.ec2.internal" podStartSLOduration=2.093854012 podStartE2EDuration="2.093854012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:25.093843202 +0000 UTC m=+3.481190641" watchObservedRunningTime="2026-07-17 13:45:25.093854012 +0000 UTC m=+3.481201451"
Jul 17 13:45:25.666353 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.665057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:25.666353 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.665111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:25.666353 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.665261    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:25.666353 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.665339    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:45:27.665321221 +0000 UTC m=+6.052668644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:25.666353 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.665705    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:25.666353 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.665751    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret podName:2d5310a1-8175-4478-b1c4-28ca2399c86b nodeName:}" failed. No retries permitted until 2026-07-17 13:45:27.665737171 +0000 UTC m=+6.053084596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret") pod "global-pull-secret-syncer-hq7zk" (UID: "2d5310a1-8175-4478-b1c4-28ca2399c86b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:25.765690 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.765632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:25.765893 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.765875    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 13:45:25.765969 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.765900    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 13:45:25.765969 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.765912    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4wrzc for pod openshift-network-diagnostics/network-check-target-cpj75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:25.766059 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:25.765976    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc podName:c0c4f750-ba8d-4262-8697-7369537dcd67 nodeName:}" failed. No retries permitted until 2026-07-17 13:45:27.765958448 +0000 UTC m=+6.153305882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4wrzc" (UniqueName: "kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc") pod "network-check-target-cpj75" (UID: "c0c4f750-ba8d-4262-8697-7369537dcd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:25.998123 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.997951    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 13:40:23 +0000 UTC" deadline="2028-01-01 07:09:18.18521143 +0000 UTC"
Jul 17 13:45:25.998123 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:25.997989    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12785h23m52.187227435s"
Jul 17 13:45:26.072119 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:26.070419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:26.072119 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:26.070551    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:26.120352 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:26.120319    2575 generic.go:356] "Generic (PLEG): container finished" podID="606cd06d04278e78e7756c2d28e16f16" containerID="98914c7c168541203e876f3781608b88d4373c31fc612accb947f9028a58db0f" exitCode=0
Jul 17 13:45:26.120820 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:26.120796    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal" event={"ID":"606cd06d04278e78e7756c2d28e16f16","Type":"ContainerDied","Data":"98914c7c168541203e876f3781608b88d4373c31fc612accb947f9028a58db0f"}
Jul 17 13:45:27.069608 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:27.068926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:27.069608 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.069056    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:27.069608 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:27.069424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:27.069608 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.069514    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:27.135318 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:27.134714    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal" event={"ID":"606cd06d04278e78e7756c2d28e16f16","Type":"ContainerStarted","Data":"1ea9f38f67921e1bf89402fd20aea42ea9d01994c6a930ea061f364fb8c3bac3"}
Jul 17 13:45:27.683549 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:27.683506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:27.683726 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:27.683564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:27.683806 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.683748    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:27.683862 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.683827    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:45:31.683810534 +0000 UTC m=+10.071157961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:27.684302 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.684234    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:27.684400 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.684306    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret podName:2d5310a1-8175-4478-b1c4-28ca2399c86b nodeName:}" failed. No retries permitted until 2026-07-17 13:45:31.684291749 +0000 UTC m=+10.071639174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret") pod "global-pull-secret-syncer-hq7zk" (UID: "2d5310a1-8175-4478-b1c4-28ca2399c86b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:27.785168 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:27.784514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:27.785168 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.784711    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 13:45:27.785168 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.784735    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 13:45:27.785168 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.784748    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4wrzc for pod openshift-network-diagnostics/network-check-target-cpj75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:27.785168 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:27.784818    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc podName:c0c4f750-ba8d-4262-8697-7369537dcd67 nodeName:}" failed. No retries permitted until 2026-07-17 13:45:31.784799096 +0000 UTC m=+10.172146528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4wrzc" (UniqueName: "kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc") pod "network-check-target-cpj75" (UID: "c0c4f750-ba8d-4262-8697-7369537dcd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:28.069098 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:28.068945    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:28.069098 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:28.069078    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:29.069139 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:29.069091    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:29.069139 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:29.069115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:29.069625 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:29.069227    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:29.069625 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:29.069337    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:30.069012 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:30.068564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:30.069012 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:30.068691    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:31.069011 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:31.068955    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:31.069564 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.069056    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:31.069564 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:31.069330    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:31.069564 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.069387    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:31.715716 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:31.715677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:31.715909 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:31.715737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:31.715978 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.715933    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:31.716042 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.715995    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:45:39.715976895 +0000 UTC m=+18.103324329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:31.716423 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.716401    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:31.716423 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.716459    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret podName:2d5310a1-8175-4478-b1c4-28ca2399c86b nodeName:}" failed. No retries permitted until 2026-07-17 13:45:39.71644432 +0000 UTC m=+18.103791752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret") pod "global-pull-secret-syncer-hq7zk" (UID: "2d5310a1-8175-4478-b1c4-28ca2399c86b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:31.817028 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:31.816896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:31.817196 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.817059    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 13:45:31.817196 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.817081    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 13:45:31.817196 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.817093    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4wrzc for pod openshift-network-diagnostics/network-check-target-cpj75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:31.817196 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:31.817156    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc podName:c0c4f750-ba8d-4262-8697-7369537dcd67 nodeName:}" failed. No retries permitted until 2026-07-17 13:45:39.817137752 +0000 UTC m=+18.204485189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4wrzc" (UniqueName: "kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc") pod "network-check-target-cpj75" (UID: "c0c4f750-ba8d-4262-8697-7369537dcd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:32.069564 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:32.069325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:32.069564 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:32.069458    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:33.068865 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:33.068788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:33.069017 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:33.068916    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:33.069084 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:33.069025    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:33.069154 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:33.069131    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:34.068678 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:34.068647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:34.069100 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:34.068759    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:35.069442 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:35.069413    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:35.069781 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:35.069510    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:35.069781 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:35.069585    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:35.069781 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:35.069666    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:36.068592 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:36.068562    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:36.068789 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:36.068709    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:37.068839 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:37.068802    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:37.069264 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:37.068923    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:37.069264 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:37.068980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:37.069264 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:37.069084    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:38.069623 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:38.069586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:38.070058 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:38.069717    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:39.069222 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:39.069142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:39.069469 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.069304    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:39.069469 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:39.069341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:39.069469 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.069456    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:39.779256 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:39.779219    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:39.779777 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:39.779288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:39.779777 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.779411    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:39.779777 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.779415    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:39.779777 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.779474    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:45:55.77945557 +0000 UTC m=+34.166802994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:39.779777 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.779491    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret podName:2d5310a1-8175-4478-b1c4-28ca2399c86b nodeName:}" failed. No retries permitted until 2026-07-17 13:45:55.779482884 +0000 UTC m=+34.166830309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret") pod "global-pull-secret-syncer-hq7zk" (UID: "2d5310a1-8175-4478-b1c4-28ca2399c86b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:39.879772 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:39.879739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:39.879956 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.879876    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 13:45:39.879956 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.879898    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 13:45:39.879956 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.879910    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4wrzc for pod openshift-network-diagnostics/network-check-target-cpj75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:39.880110 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:39.879964    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc podName:c0c4f750-ba8d-4262-8697-7369537dcd67 nodeName:}" failed. No retries permitted until 2026-07-17 13:45:55.879947556 +0000 UTC m=+34.267294974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4wrzc" (UniqueName: "kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc") pod "network-check-target-cpj75" (UID: "c0c4f750-ba8d-4262-8697-7369537dcd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:40.069559 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:40.069476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:40.069721 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:40.069595    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:41.069407 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:41.069374    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:41.069820 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:41.069374    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:41.069820 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:41.069503    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:41.069820 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:41.069524    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:42.070101 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:42.070072    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:42.070589 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:42.070163    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:43.068948 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:43.068923    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:43.069055 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:43.068923    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:43.069055 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:43.069020    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:43.069142 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:43.069096    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:44.069574 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.069349    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:44.070463 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:44.069630    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:44.167905 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.167862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gvl6k" event={"ID":"c74d1625-9ecb-45cd-bc27-3f87ed8d40b3","Type":"ContainerStarted","Data":"2c77a70ed12e5fb20bbca3fa0825bacb805ef44e792e4011a0a6c351fed4e0ff"}
Jul 17 13:45:44.169092 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.169072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58ggv" event={"ID":"ef9b353f-aeb1-4d0b-adfb-01e472913c4b","Type":"ContainerStarted","Data":"8caf854b5fd08aa44bdd54a60bc768f52519844218eb7c5a7932f73afc1c85af"}
Jul 17 13:45:44.170384 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.170367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb" event={"ID":"060fa2ff-277d-4d2a-9a45-533cc45fdb8a","Type":"ContainerStarted","Data":"d9313d1925e2431de46e02668ec99317a63f708c138e99f44eb83c3a336de5aa"}
Jul 17 13:45:44.171838 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.171818    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8c704e5-f773-48ec-9d76-717630b9aecb" containerID="f0717c34a909cde7b743096f04dc4e6624180dfb38c201fd1682bd7996c7e430" exitCode=0
Jul 17 13:45:44.171913 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.171892    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerDied","Data":"f0717c34a909cde7b743096f04dc4e6624180dfb38c201fd1682bd7996c7e430"}
Jul 17 13:45:44.174623 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.174609    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:45:44.174906 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.174888    2575 generic.go:356] "Generic (PLEG): container finished" podID="13dc9407-48bb-4162-b663-f642fb0a6248" containerID="f631c91762d135f410827ac48c407020832cf743a29793597549cdd73021c94b" exitCode=1
Jul 17 13:45:44.174968 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.174950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"2336def03eeaa1833e089ba1fb6cf9d50502866590717e88d46a5fa433e42f1c"}
Jul 17 13:45:44.175041 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.175027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"54bc291bdaaccf9cf837822f232508d98e83bfe7ef311c9331b9bc2179f50e8e"}
Jul 17 13:45:44.175094 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.175080    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"449816af68b6a2b232c2a780bf32d4a66d182d7b6a3c08cc12dcd471ed25377b"}
Jul 17 13:45:44.175126 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.175119    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"10268e1776d81227aa0d6040c336569c96a5c06807391635888130eb9a0e4aca"}
Jul 17 13:45:44.175164 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.175146    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerDied","Data":"f631c91762d135f410827ac48c407020832cf743a29793597549cdd73021c94b"}
Jul 17 13:45:44.175209 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.175173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"0ee68d93d0256cab29d1c37de4a4462311a901b03ba4e8eed35da53ca6f624fc"}
Jul 17 13:45:44.176128 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.176098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t47gn" event={"ID":"b3405f65-f7c9-4913-8d29-611a74961d64","Type":"ContainerStarted","Data":"eb96abaf2b4f22fd4e3d2d89526a1be3d6c0c77ed1fae4669bba5debc54c3d80"}
Jul 17 13:45:44.177387 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.177368    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tj2bb" event={"ID":"61c8b7ed-7cda-4535-9499-9bfe558d39c0","Type":"ContainerStarted","Data":"abc968582ab4c6db8e9c114e40afaa15bc117874997c77e140ba15538748d1df"}
Jul 17 13:45:44.178704 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.178686    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4xsq" event={"ID":"85d482fb-3510-4df0-8302-71ca87962c0b","Type":"ContainerStarted","Data":"8ae676b41c80f504659242af2d10ac21933fd801041e5c070ed01167b6319cf2"}
Jul 17 13:45:44.184408 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.184376    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-191.ec2.internal" podStartSLOduration=21.184364428 podStartE2EDuration="21.184364428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:27.152909147 +0000 UTC m=+5.540256588" watchObservedRunningTime="2026-07-17 13:45:44.184364428 +0000 UTC m=+22.571711868"
Jul 17 13:45:44.184787 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.184765    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gvl6k" podStartSLOduration=4.021546479 podStartE2EDuration="22.184759384s" totalImagesPullingTime="18.163212905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:44.184602849 +0000 UTC m=+22.571950271" watchObservedRunningTime="2026-07-17 13:45:44.184759384 +0000 UTC m=+22.572106824"
Jul 17 13:45:44.231532 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.231501    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tj2bb" podStartSLOduration=3.600625591 podStartE2EDuration="22.231490425s" totalImagesPullingTime="18.630864834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:44.204727522 +0000 UTC m=+22.592074962" watchObservedRunningTime="2026-07-17 13:45:44.231490425 +0000 UTC m=+22.618837865"
Jul 17 13:45:44.252936 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.252907    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t47gn" podStartSLOduration=8.499781933 podStartE2EDuration="22.252896401s" totalImagesPullingTime="13.753114468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:44.251957441 +0000 UTC m=+22.639304880" watchObservedRunningTime="2026-07-17 13:45:44.252896401 +0000 UTC m=+22.640243841"
Jul 17 13:45:44.301106 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.301073    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q4xsq" podStartSLOduration=4.141289042 podStartE2EDuration="22.301063063s" totalImagesPullingTime="18.159774021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:44.27396794 +0000 UTC m=+22.661315380" watchObservedRunningTime="2026-07-17 13:45:44.301063063 +0000 UTC m=+22.688410502"
Jul 17 13:45:44.506711 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:44.506688    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 13:45:45.030806 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.030644    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T13:45:44.506708411Z","UUID":"28bfb70e-2ded-47f0-8702-bd4ff81270e9","Handler":null,"Name":"","Endpoint":""}
Jul 17 13:45:45.032811 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.032552    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 13:45:45.032811 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.032603    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 13:45:45.069664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.069629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:45.070090 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:45.069768    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:45.070090 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.069861    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:45.070090 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:45.069943    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:45.182806 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.182585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wb66s" event={"ID":"39d27342-9a1d-4bc9-befe-1f2c30b84406","Type":"ContainerStarted","Data":"ccba73ab5864f7bdcabb03491353619e704257d4a6fdb0fd42541aafaa30532d"}
Jul 17 13:45:45.184925 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.184874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb" event={"ID":"060fa2ff-277d-4d2a-9a45-533cc45fdb8a","Type":"ContainerStarted","Data":"dce36122453604256c9a6dbdaf33a68bbfc5e47d3893d35c3ca06f36784db2b2"}
Jul 17 13:45:45.200289 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.200227    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-58ggv" podStartSLOduration=5.04115819 podStartE2EDuration="23.200208327s" totalImagesPullingTime="18.159050137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:44.301193633 +0000 UTC m=+22.688541073" watchObservedRunningTime="2026-07-17 13:45:45.200208327 +0000 UTC m=+23.587555779"
Jul 17 13:45:45.201138 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:45.201097    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wb66s" podStartSLOduration=5.031003253 podStartE2EDuration="23.201081766s" totalImagesPullingTime="18.170078513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:45.200029685 +0000 UTC m=+23.587377180" watchObservedRunningTime="2026-07-17 13:45:45.201081766 +0000 UTC m=+23.588429207"
Jul 17 13:45:46.069582 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:46.069319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:46.069582 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:46.069453    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:46.189679 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:46.189585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb" event={"ID":"060fa2ff-277d-4d2a-9a45-533cc45fdb8a","Type":"ContainerStarted","Data":"fc236f4585cdda9f5b02f30bfce8dc161a5e5c92d3fa28d6741662900a7ac707"}
Jul 17 13:45:46.192738 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:46.192718    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:45:46.193137 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:46.193098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"210f28d3cc878c424ea5e243e0b9bfac811dcfcfc2f49f185adfa9edc1f86e4b"}
Jul 17 13:45:47.069193 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:47.069165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:47.069368 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:47.069305    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:47.069368 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:47.069357    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:47.069494 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:47.069456    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:47.807545 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:47.807510    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:47.808219 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:47.808202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:47.833102 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:47.833053    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj7qb" podStartSLOduration=5.58057499 podStartE2EDuration="25.833039365s" totalImagesPullingTime="20.252464375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:46.217057788 +0000 UTC m=+24.604405228" watchObservedRunningTime="2026-07-17 13:45:47.833039365 +0000 UTC m=+26.220386827"
Jul 17 13:45:48.069257 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.069192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:48.069421 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:48.069300    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:48.201723 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.201575    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:45:48.202329 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.202131    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"a7ce61a4082be42923b7bb1c96170055bf14ea9df54fdcb78fa95fffd4e413c8"}
Jul 17 13:45:48.203676 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.202593    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:48.203676 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.202669    2575 scope.go:117] "RemoveContainer" containerID="f631c91762d135f410827ac48c407020832cf743a29793597549cdd73021c94b"
Jul 17 13:45:48.203676 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.202973    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:48.203676 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.203053    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:48.203676 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.203098    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:48.203676 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.203302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t47gn"
Jul 17 13:45:48.220448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.220198    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:48.221222 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:48.221206    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:45:49.069453 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:49.069428    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:49.069977 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:49.069432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:49.069977 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:49.069519    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:49.069977 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:49.069596    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:49.205772 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:49.205747    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8c704e5-f773-48ec-9d76-717630b9aecb" containerID="3c7bacda0d968687708dfc721a90f830f5cb2239a7fe58d990acee5d88e79b18" exitCode=0
Jul 17 13:45:49.205899 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:49.205832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerDied","Data":"3c7bacda0d968687708dfc721a90f830f5cb2239a7fe58d990acee5d88e79b18"}
Jul 17 13:45:49.209898 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:49.209882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:45:49.210240 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:49.210211    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" event={"ID":"13dc9407-48bb-4162-b663-f642fb0a6248","Type":"ContainerStarted","Data":"82feac10502e6251520ff92a7d0abe13acf35e22ab27b56f2f28c330ae021d25"}
Jul 17 13:45:50.069582 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.069466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:50.070054 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:50.069658    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:50.193452 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.193407    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc" podStartSLOduration=9.989015034 podStartE2EDuration="28.193391362s" totalImagesPullingTime="18.204376328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:49.265635985 +0000 UTC m=+27.652983426" watchObservedRunningTime="2026-07-17 13:45:50.193391362 +0000 UTC m=+28.580738800"
Jul 17 13:45:50.194102 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.194084    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cr2qc"]
Jul 17 13:45:50.194183 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.194173    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:50.194270 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:50.194255    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:50.198999 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.198979    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cpj75"]
Jul 17 13:45:50.199073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.199051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:50.199138 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:50.199122    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:50.204848 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.204789    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hq7zk"]
Jul 17 13:45:50.215205 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.215184    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8c704e5-f773-48ec-9d76-717630b9aecb" containerID="bef48f5fa6f46e47ac73bf6bf3981224727e9e47e6c39a39505fc8638ee080a6" exitCode=0
Jul 17 13:45:50.215311 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.215255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:50.215311 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:50.215298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerDied","Data":"bef48f5fa6f46e47ac73bf6bf3981224727e9e47e6c39a39505fc8638ee080a6"}
Jul 17 13:45:50.216193 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:50.215876    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:52.069883 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:52.069850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:52.070402 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:52.069939    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:52.070402 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:52.069945    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:52.070402 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:52.069978    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:52.070402 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:52.070016    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:52.070402 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:52.070071    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:52.222007 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:52.221979    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8c704e5-f773-48ec-9d76-717630b9aecb" containerID="908fb3436032ad2a6baa9218ceb3197c86f2b2d6558ce57c9b08d0ec784c229f" exitCode=0
Jul 17 13:45:52.222130 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:52.222024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerDied","Data":"908fb3436032ad2a6baa9218ceb3197c86f2b2d6558ce57c9b08d0ec784c229f"}
Jul 17 13:45:54.069063 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:54.068997    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:54.069792 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:54.068995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:54.069792 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:54.069126    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:54.069792 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:54.069213    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:54.069792 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:54.068995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:54.069792 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:54.069340    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:55.800878 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:55.800841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:55.800878 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:55.800884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:55.801334 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.801011    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:55.801334 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.801065    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:46:27.801048821 +0000 UTC m=+66.188396239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 13:45:55.801334 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.801009    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:55.801334 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.801169    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret podName:2d5310a1-8175-4478-b1c4-28ca2399c86b nodeName:}" failed. No retries permitted until 2026-07-17 13:46:27.801141006 +0000 UTC m=+66.188488438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret") pod "global-pull-secret-syncer-hq7zk" (UID: "2d5310a1-8175-4478-b1c4-28ca2399c86b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 13:45:55.902074 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:55.902035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:55.902244 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.902209    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 13:45:55.902244 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.902232    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 13:45:55.902356 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.902248    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4wrzc for pod openshift-network-diagnostics/network-check-target-cpj75: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:55.902356 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:55.902317    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc podName:c0c4f750-ba8d-4262-8697-7369537dcd67 nodeName:}" failed. No retries permitted until 2026-07-17 13:46:27.902303332 +0000 UTC m=+66.289650776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4wrzc" (UniqueName: "kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc") pod "network-check-target-cpj75" (UID: "c0c4f750-ba8d-4262-8697-7369537dcd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 13:45:56.069647 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.069449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:56.069748 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.069516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:56.069748 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:56.069739    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq7zk" podUID="2d5310a1-8175-4478-b1c4-28ca2399c86b"
Jul 17 13:45:56.069869 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.069516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:56.069869 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:56.069821    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpj75" podUID="c0c4f750-ba8d-4262-8697-7369537dcd67"
Jul 17 13:45:56.069972 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:56.069952    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:45:56.234617 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.234591    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8c704e5-f773-48ec-9d76-717630b9aecb" containerID="6e8d6e7034d65d56f71cfce0f1fa59e6eaf226c97abae28aabbf269076720c3a" exitCode=0
Jul 17 13:45:56.234715 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.234653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerDied","Data":"6e8d6e7034d65d56f71cfce0f1fa59e6eaf226c97abae28aabbf269076720c3a"}
Jul 17 13:45:56.427911 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.427889    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-191.ec2.internal" event="NodeReady"
Jul 17 13:45:56.428045 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.428003    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 13:45:56.480546 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.480513    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g7wdl"]
Jul 17 13:45:56.482345 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.482330    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5x48l"]
Jul 17 13:45:56.482471 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.482456    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.483928 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.483909    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.484729 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.484707    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 13:45:56.484830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.484815    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 13:45:56.485045 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.485030    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 13:45:56.485105 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.485034    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v8874\""
Jul 17 13:45:56.485793 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.485776    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 13:45:56.485793 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.485787    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 13:45:56.485909 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.485826    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gl8vw\""
Jul 17 13:45:56.495015 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.494997    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5x48l"]
Jul 17 13:45:56.495737 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.495721    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g7wdl"]
Jul 17 13:45:56.606794 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.606734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.606794 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.606776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d686534-59bf-492c-89e5-98689ba395df-tmp-dir\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.606794 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.606796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e1a49fa-42d8-4f65-a8a2-7497d017e06d-cert\") pod \"ingress-canary-g7wdl\" (UID: \"0e1a49fa-42d8-4f65-a8a2-7497d017e06d\") " pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.606980 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.606845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4528g\" (UniqueName: \"kubernetes.io/projected/2d686534-59bf-492c-89e5-98689ba395df-kube-api-access-4528g\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.606980 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.606903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ln65\" (UniqueName: \"kubernetes.io/projected/0e1a49fa-42d8-4f65-a8a2-7497d017e06d-kube-api-access-5ln65\") pod \"ingress-canary-g7wdl\" (UID: \"0e1a49fa-42d8-4f65-a8a2-7497d017e06d\") " pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.606980 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.606936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d686534-59bf-492c-89e5-98689ba395df-config-volume\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.707522 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.707496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d686534-59bf-492c-89e5-98689ba395df-tmp-dir\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.707634 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.707528    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e1a49fa-42d8-4f65-a8a2-7497d017e06d-cert\") pod \"ingress-canary-g7wdl\" (UID: \"0e1a49fa-42d8-4f65-a8a2-7497d017e06d\") " pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.707634 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.707543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4528g\" (UniqueName: \"kubernetes.io/projected/2d686534-59bf-492c-89e5-98689ba395df-kube-api-access-4528g\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.707728 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.707677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ln65\" (UniqueName: \"kubernetes.io/projected/0e1a49fa-42d8-4f65-a8a2-7497d017e06d-kube-api-access-5ln65\") pod \"ingress-canary-g7wdl\" (UID: \"0e1a49fa-42d8-4f65-a8a2-7497d017e06d\") " pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.707728 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.707720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d686534-59bf-492c-89e5-98689ba395df-config-volume\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.707835 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.707747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.707881 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:56.707836    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:45:56.707922 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:56.707883    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:45:57.207870642 +0000 UTC m=+35.595218063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:45:56.707978 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.707833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d686534-59bf-492c-89e5-98689ba395df-tmp-dir\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.708151 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.708134    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d686534-59bf-492c-89e5-98689ba395df-config-volume\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.711702 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.711681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e1a49fa-42d8-4f65-a8a2-7497d017e06d-cert\") pod \"ingress-canary-g7wdl\" (UID: \"0e1a49fa-42d8-4f65-a8a2-7497d017e06d\") " pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.719832 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.719813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4528g\" (UniqueName: \"kubernetes.io/projected/2d686534-59bf-492c-89e5-98689ba395df-kube-api-access-4528g\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:56.719913 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.719847    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ln65\" (UniqueName: \"kubernetes.io/projected/0e1a49fa-42d8-4f65-a8a2-7497d017e06d-kube-api-access-5ln65\") pod \"ingress-canary-g7wdl\" (UID: \"0e1a49fa-42d8-4f65-a8a2-7497d017e06d\") " pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.792883 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.792856    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g7wdl"
Jul 17 13:45:56.917214 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:56.917190    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g7wdl"]
Jul 17 13:45:56.920499 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:45:56.920476    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1a49fa_42d8_4f65_a8a2_7497d017e06d.slice/crio-9e97fa7c5ab58ccb57179db0734fa018c41a4337732ea7e33bfc97692f05172e WatchSource:0}: Error finding container 9e97fa7c5ab58ccb57179db0734fa018c41a4337732ea7e33bfc97692f05172e: Status 404 returned error can't find the container with id 9e97fa7c5ab58ccb57179db0734fa018c41a4337732ea7e33bfc97692f05172e
Jul 17 13:45:57.211146 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:57.211067    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:57.211298 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:57.211204    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:45:57.211298 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:57.211256    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:45:58.211242915 +0000 UTC m=+36.598590333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:45:57.237801 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:57.237766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g7wdl" event={"ID":"0e1a49fa-42d8-4f65-a8a2-7497d017e06d","Type":"ContainerStarted","Data":"9e97fa7c5ab58ccb57179db0734fa018c41a4337732ea7e33bfc97692f05172e"}
Jul 17 13:45:57.240725 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:57.240700    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8c704e5-f773-48ec-9d76-717630b9aecb" containerID="3bb079b71c36c084375f866bbdab920d16c912beeaf4f4719fc50f29d2e29358" exitCode=0
Jul 17 13:45:57.240846 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:57.240740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerDied","Data":"3bb079b71c36c084375f866bbdab920d16c912beeaf4f4719fc50f29d2e29358"}
Jul 17 13:45:58.069448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.069255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:45:58.069887 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.069254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:45:58.069887 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.069325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:45:58.073092 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.073066    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 13:45:58.073225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.073099    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9xnsb\""
Jul 17 13:45:58.073225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.073124    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 13:45:58.073225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.073152    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 13:45:58.073225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.073106    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tpk25\""
Jul 17 13:45:58.073453 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.073436    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 13:45:58.217372 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.217336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:45:58.217542 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:58.217414    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:45:58.217542 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:45:58.217479    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:46:00.217461026 +0000 UTC m=+38.604808464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:45:58.248448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:45:58.248412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88s54" event={"ID":"d8c704e5-f773-48ec-9d76-717630b9aecb","Type":"ContainerStarted","Data":"ac79cce8ca26ea1eaa55051d47d56027fc7f625f9e11566bc75de99b67a9d8eb"}
Jul 17 13:46:00.231362 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:00.231337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:46:00.231866 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:00.231468    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:46:00.231866 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:00.231523    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:46:04.231510086 +0000 UTC m=+42.618857504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:46:00.253695 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:00.253666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g7wdl" event={"ID":"0e1a49fa-42d8-4f65-a8a2-7497d017e06d","Type":"ContainerStarted","Data":"eb94a0a6d910b31026daaf8822bbfc060eb1e954b3a7bad0f14361c2f8eb0253"}
Jul 17 13:46:00.274209 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:00.274165    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-88s54" podStartSLOduration=10.433090501 podStartE2EDuration="38.274151712s" totalImagesPullingTime="27.841061211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:45:58.274200786 +0000 UTC m=+36.661548318" watchObservedRunningTime="2026-07-17 13:46:00.274151712 +0000 UTC m=+38.661499151"
Jul 17 13:46:04.258524 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:04.258484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:46:04.258996 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:04.258638    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:46:04.258996 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:04.258699    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:46:12.258683229 +0000 UTC m=+50.646030647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:46:12.316781 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:12.316750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:46:12.317215 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:12.316864    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:46:12.317215 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:12.316922    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:46:28.316909116 +0000 UTC m=+66.704256534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:46:20.231216 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:20.231186    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhqvc"
Jul 17 13:46:20.258996 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:20.258945    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g7wdl" podStartSLOduration=22.007048895 podStartE2EDuration="24.258930981s" totalImagesPullingTime="2.251882086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:46:00.274925129 +0000 UTC m=+38.662272567" watchObservedRunningTime="2026-07-17 13:46:20.258930981 +0000 UTC m=+58.646278448"
Jul 17 13:46:27.824171 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.824137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:46:27.824171 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.824183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:46:27.826552 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.826529    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 13:46:27.826640 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.826624    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 13:46:27.834553 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:27.834537    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 13:46:27.834629 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:27.834586    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:47:31.834572223 +0000 UTC m=+130.221919641 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : secret "metrics-daemon-secret" not found
Jul 17 13:46:27.837736 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.837717    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d5310a1-8175-4478-b1c4-28ca2399c86b-original-pull-secret\") pod \"global-pull-secret-syncer-hq7zk\" (UID: \"2d5310a1-8175-4478-b1c4-28ca2399c86b\") " pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:46:27.925352 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.925324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:46:27.927632 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.927616    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 13:46:27.938544 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.938524    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 13:46:27.949573 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:27.949549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wrzc\" (UniqueName: \"kubernetes.io/projected/c0c4f750-ba8d-4262-8697-7369537dcd67-kube-api-access-4wrzc\") pod \"network-check-target-cpj75\" (UID: \"c0c4f750-ba8d-4262-8697-7369537dcd67\") " pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:46:28.087642 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.087582    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq7zk"
Jul 17 13:46:28.094884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.094861    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tpk25\""
Jul 17 13:46:28.103591 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.103567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:46:28.246827 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.246796    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hq7zk"]
Jul 17 13:46:28.249997 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:46:28.249967    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5310a1_8175_4478_b1c4_28ca2399c86b.slice/crio-3d2d64457787543a7cdcecbc5d50e148c8312b32937bf45ec0958c69857c3f8b WatchSource:0}: Error finding container 3d2d64457787543a7cdcecbc5d50e148c8312b32937bf45ec0958c69857c3f8b: Status 404 returned error can't find the container with id 3d2d64457787543a7cdcecbc5d50e148c8312b32937bf45ec0958c69857c3f8b
Jul 17 13:46:28.261891 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.261855    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cpj75"]
Jul 17 13:46:28.276501 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:46:28.276476    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c4f750_ba8d_4262_8697_7369537dcd67.slice/crio-cf5520c9dc535c771081fa069da25059503f6cd2e48e3423dc65bed86f010825 WatchSource:0}: Error finding container cf5520c9dc535c771081fa069da25059503f6cd2e48e3423dc65bed86f010825: Status 404 returned error can't find the container with id cf5520c9dc535c771081fa069da25059503f6cd2e48e3423dc65bed86f010825
Jul 17 13:46:28.320588 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.320550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cpj75" event={"ID":"c0c4f750-ba8d-4262-8697-7369537dcd67","Type":"ContainerStarted","Data":"cf5520c9dc535c771081fa069da25059503f6cd2e48e3423dc65bed86f010825"}
Jul 17 13:46:28.321516 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.321494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hq7zk" event={"ID":"2d5310a1-8175-4478-b1c4-28ca2399c86b","Type":"ContainerStarted","Data":"3d2d64457787543a7cdcecbc5d50e148c8312b32937bf45ec0958c69857c3f8b"}
Jul 17 13:46:28.327819 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:28.327801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:46:28.327943 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:28.327928    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:46:28.328002 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:46:28.327994    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:47:00.327974168 +0000 UTC m=+98.715321589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:46:33.335589 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:33.335544    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cpj75" event={"ID":"c0c4f750-ba8d-4262-8697-7369537dcd67","Type":"ContainerStarted","Data":"8a974ef69f1a6bf2b545603fda4dab34ca427aa407ccfb450e634d9b86ac83b1"}
Jul 17 13:46:33.336065 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:33.335667    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:46:33.337393 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:33.337366    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hq7zk" event={"ID":"2d5310a1-8175-4478-b1c4-28ca2399c86b","Type":"ContainerStarted","Data":"99082751e4e98aa050c3134a1c8460294cd57586eb77e9db282dd5c0f2fd4868"}
Jul 17 13:46:33.353622 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:33.353569    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cpj75" podStartSLOduration=66.990700995 podStartE2EDuration="1m11.353534797s" totalImagesPullingTime="4.362833802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:46:33.351768571 +0000 UTC m=+71.739116011" watchObservedRunningTime="2026-07-17 13:46:33.353534797 +0000 UTC m=+71.740882215"
Jul 17 13:46:33.367347 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:46:33.367299    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hq7zk" podStartSLOduration=66.9741404 podStartE2EDuration="1m11.367259648s" totalImagesPullingTime="4.393119248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:46:33.367019978 +0000 UTC m=+71.754367417" watchObservedRunningTime="2026-07-17 13:46:33.367259648 +0000 UTC m=+71.754607085"
Jul 17 13:47:00.426226 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:47:00.426189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:47:00.426686 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:47:00.426322    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:47:00.426686 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:47:00.426386    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:48:04.426372575 +0000 UTC m=+162.813719994 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:47:04.342886 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:47:04.342849    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cpj75"
Jul 17 13:47:31.927031 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:47:31.926990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:47:31.927582 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:47:31.927159    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 13:47:31.927582 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:47:31.927248    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:49:33.92722698 +0000 UTC m=+252.314574400 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : secret "metrics-daemon-secret" not found
Jul 17 13:47:59.498069 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:47:59.498020    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5x48l" podUID="2d686534-59bf-492c-89e5-98689ba395df"
Jul 17 13:47:59.533120 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:47:59.533090    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x48l"
Jul 17 13:48:01.080985 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:48:01.080946    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:48:04.426570 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:48:04.426539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:48:04.426934 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:48:04.426660    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 13:48:04.426934 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:48:04.426710    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls podName:2d686534-59bf-492c-89e5-98689ba395df nodeName:}" failed. No retries permitted until 2026-07-17 13:50:06.426696487 +0000 UTC m=+284.814043905 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls") pod "dns-default-5x48l" (UID: "2d686534-59bf-492c-89e5-98689ba395df") : secret "dns-default-metrics-tls" not found
Jul 17 13:48:16.069217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:48:16.069144    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:49:33.996262 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:33.996223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:49:33.996725 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:33.996409    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 13:49:33.996725 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:33.996486    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs podName:8ec62e35-b2e6-4161-871f-3463c827961d nodeName:}" failed. No retries permitted until 2026-07-17 13:51:35.996466711 +0000 UTC m=+374.383814131 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs") pod "network-metrics-daemon-cr2qc" (UID: "8ec62e35-b2e6-4161-871f-3463c827961d") : secret "metrics-daemon-secret" not found
Jul 17 13:49:36.731065 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.731024    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"]
Jul 17 13:49:36.733120 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.733101    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.737602 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.737567    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 17 13:49:36.737786 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.737660    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 13:49:36.737786 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.737675    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fmdgx\""
Jul 17 13:49:36.738451 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.738423    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 13:49:36.739525 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.739497    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 17 13:49:36.772291 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.772258    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"]
Jul 17 13:49:36.814791 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.814768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8hk\" (UniqueName: \"kubernetes.io/projected/ba0000c9-84a7-4f61-87e0-69d14de53c10-kube-api-access-cx8hk\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.814884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.814804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.814884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.814847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba0000c9-84a7-4f61-87e0-69d14de53c10-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.824899 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.824876    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"]
Jul 17 13:49:36.826644 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.826628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:36.830000 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.829980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 17 13:49:36.830610 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.830597    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 17 13:49:36.834291 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.834264    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 17 13:49:36.834420 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.834406    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z7grb\""
Jul 17 13:49:36.848674 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.848654    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wmcnx"]
Jul 17 13:49:36.850241 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.850223    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"]
Jul 17 13:49:36.850327 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.850314    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:36.852924 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.852783    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 17 13:49:36.853569 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.853547    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pm9g6\""
Jul 17 13:49:36.853569 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.853560    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"]
Jul 17 13:49:36.853809 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.853788    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 17 13:49:36.853909 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.853830    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 17 13:49:36.853909 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.853868    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 17 13:49:36.855377 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.855359    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"
Jul 17 13:49:36.858110 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.858063    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 17 13:49:36.858310 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.858293    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vjt7g\""
Jul 17 13:49:36.858468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.858456    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 17 13:49:36.864337 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.861229    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 17 13:49:36.869391 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.869369    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wmcnx"]
Jul 17 13:49:36.875953 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.875934    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"]
Jul 17 13:49:36.915066 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.915158 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-config\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:36.915158 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-serving-cert\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:36.915158 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-trusted-ca\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:36.915269 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlcn\" (UniqueName: \"kubernetes.io/projected/9a868e7f-3f04-469f-8fe4-551bda323cf1-kube-api-access-rjlcn\") pod \"volume-data-source-validator-78cf4c6f6-spqnn\" (UID: \"9a868e7f-3f04-469f-8fe4-551bda323cf1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"
Jul 17 13:49:36.915269 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:36.915180    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:36.915269 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915217    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:36.915269 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:36.915228    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls podName:ba0000c9-84a7-4f61-87e0-69d14de53c10 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:37.415216033 +0000 UTC m=+255.802563451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7r6jb" (UID: "ba0000c9-84a7-4f61-87e0-69d14de53c10") : secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:36.915269 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915262    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbjnw\" (UniqueName: \"kubernetes.io/projected/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-kube-api-access-lbjnw\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:36.915463 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba0000c9-84a7-4f61-87e0-69d14de53c10-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.915463 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8hk\" (UniqueName: \"kubernetes.io/projected/ba0000c9-84a7-4f61-87e0-69d14de53c10-kube-api-access-cx8hk\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.915463 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915358    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86jb2\" (UniqueName: \"kubernetes.io/projected/48e7128a-0b83-44df-957e-34e25e338ec0-kube-api-access-86jb2\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:36.915866 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.915848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba0000c9-84a7-4f61-87e0-69d14de53c10-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.927311 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.927288    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8hk\" (UniqueName: \"kubernetes.io/projected/ba0000c9-84a7-4f61-87e0-69d14de53c10-kube-api-access-cx8hk\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:36.933683 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.933666    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"]
Jul 17 13:49:36.935316 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.935301    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cd9c4b87-9dsn6"]
Jul 17 13:49:36.935413 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.935398    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:36.937123 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.937106    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:36.940179 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.940159    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 17 13:49:36.941159 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.941141    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gk255\""
Jul 17 13:49:36.941838 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.941819    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 17 13:49:36.941943 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.941925    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 17 13:49:36.942192 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.942176    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 17 13:49:36.942238 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.942184    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 13:49:36.944713 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.944699    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 13:49:36.944713 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.944707    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 13:49:36.944863 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.944733    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cnxqs\""
Jul 17 13:49:36.949647 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.949631    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"]
Jul 17 13:49:36.951926 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.951909    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 13:49:36.970092 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:36.970067    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd9c4b87-9dsn6"]
Jul 17 13:49:37.016089 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlcn\" (UniqueName: \"kubernetes.io/projected/9a868e7f-3f04-469f-8fe4-551bda323cf1-kube-api-access-rjlcn\") pod \"volume-data-source-validator-78cf4c6f6-spqnn\" (UID: \"9a868e7f-3f04-469f-8fe4-551bda323cf1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"
Jul 17 13:49:37.016089 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-trusted-ca\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016248 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d6ef3b-0c52-4586-a224-7943c1b53190-ca-trust-extracted\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016248 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-config\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.016248 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-serving-cert\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.016385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016263    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-image-registry-private-configuration\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016320    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-bound-sa-token\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-installation-pull-secrets\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:37.016385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbjnw\" (UniqueName: \"kubernetes.io/projected/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-kube-api-access-lbjnw\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.016588 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.016588 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.016466    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 13:49:37.016588 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.016588 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.016510    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls podName:48e7128a-0b83-44df-957e-34e25e338ec0 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:37.516496559 +0000 UTC m=+255.903843980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sv6kl" (UID: "48e7128a-0b83-44df-957e-34e25e338ec0") : secret "samples-operator-tls" not found
Jul 17 13:49:37.016588 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psp4j\" (UniqueName: \"kubernetes.io/projected/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-kube-api-access-psp4j\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.016845 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016583    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86jb2\" (UniqueName: \"kubernetes.io/projected/48e7128a-0b83-44df-957e-34e25e338ec0-kube-api-access-86jb2\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:37.016845 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016845 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-certificates\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016845 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-trusted-ca\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.016845 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gcw\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-kube-api-access-28gcw\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.016845 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.016841    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-config\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.017442 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.017426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-trusted-ca\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.018348 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.018333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-serving-cert\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.036160 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.036133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlcn\" (UniqueName: \"kubernetes.io/projected/9a868e7f-3f04-469f-8fe4-551bda323cf1-kube-api-access-rjlcn\") pod \"volume-data-source-validator-78cf4c6f6-spqnn\" (UID: \"9a868e7f-3f04-469f-8fe4-551bda323cf1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"
Jul 17 13:49:37.036271 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.036256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86jb2\" (UniqueName: \"kubernetes.io/projected/48e7128a-0b83-44df-957e-34e25e338ec0-kube-api-access-86jb2\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:37.036749 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.036732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbjnw\" (UniqueName: \"kubernetes.io/projected/e8b7a27d-388f-4596-ad2b-bf20ad40b2a0-kube-api-access-lbjnw\") pod \"console-operator-565b7b569d-wmcnx\" (UID: \"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0\") " pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.117473 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-installation-pull-secrets\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.117569 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.117569 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.117569 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psp4j\" (UniqueName: \"kubernetes.io/projected/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-kube-api-access-psp4j\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.117724 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.117724 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-certificates\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.117724 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gcw\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-kube-api-access-28gcw\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.117724 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.117677    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 13:49:37.117724 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.117691    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd9c4b87-9dsn6: secret "image-registry-tls" not found
Jul 17 13:49:37.117947 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.117744    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls podName:f8d6ef3b-0c52-4586-a224-7943c1b53190 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:37.617729059 +0000 UTC m=+256.005076493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls") pod "image-registry-7cd9c4b87-9dsn6" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190") : secret "image-registry-tls" not found
Jul 17 13:49:37.117947 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-trusted-ca\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.117947 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d6ef3b-0c52-4586-a224-7943c1b53190-ca-trust-extracted\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.117947 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-image-registry-private-configuration\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.117947 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.117890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-bound-sa-token\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.118192 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.118150    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.118251 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.118230    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d6ef3b-0c52-4586-a224-7943c1b53190-ca-trust-extracted\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.118336 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.118313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-certificates\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.118671 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.118653    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-trusted-ca\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.119724 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.119698    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-installation-pull-secrets\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.119811 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.119704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.120208 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.120189    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-image-registry-private-configuration\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.126775 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.126753    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psp4j\" (UniqueName: \"kubernetes.io/projected/d6a6e949-2fd9-49fe-9d3d-7adc6e516721-kube-api-access-psp4j\") pod \"kube-storage-version-migrator-operator-7658d9b749-n5k7m\" (UID: \"d6a6e949-2fd9-49fe-9d3d-7adc6e516721\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.126965 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.126948    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gcw\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-kube-api-access-28gcw\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.131984 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.131967    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-bound-sa-token\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.161200 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.161182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:37.168517 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.168497    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"
Jul 17 13:49:37.244743 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.244716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"
Jul 17 13:49:37.295884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.295809    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wmcnx"]
Jul 17 13:49:37.299143 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:49:37.299116    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b7a27d_388f_4596_ad2b_bf20ad40b2a0.slice/crio-a17a216952152f8456ef03d29ee3304f78d0f9c66ab2e19ba2ad44b0dff1026d WatchSource:0}: Error finding container a17a216952152f8456ef03d29ee3304f78d0f9c66ab2e19ba2ad44b0dff1026d: Status 404 returned error can't find the container with id a17a216952152f8456ef03d29ee3304f78d0f9c66ab2e19ba2ad44b0dff1026d
Jul 17 13:49:37.311654 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.311631    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn"]
Jul 17 13:49:37.373993 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.373955    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m"]
Jul 17 13:49:37.376862 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:49:37.376840    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a6e949_2fd9_49fe_9d3d_7adc6e516721.slice/crio-5144dd65cdca8ef27db0c47f97a466b28e4e6778abcc8f3e1e573e5735557ca0 WatchSource:0}: Error finding container 5144dd65cdca8ef27db0c47f97a466b28e4e6778abcc8f3e1e573e5735557ca0: Status 404 returned error can't find the container with id 5144dd65cdca8ef27db0c47f97a466b28e4e6778abcc8f3e1e573e5735557ca0
Jul 17 13:49:37.419553 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.419525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:37.419655 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.419643    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:37.419694 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.419689    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls podName:ba0000c9-84a7-4f61-87e0-69d14de53c10 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:38.419676693 +0000 UTC m=+256.807024111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7r6jb" (UID: "ba0000c9-84a7-4f61-87e0-69d14de53c10") : secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:37.520136 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.520109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:37.520299 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.520261    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 13:49:37.520377 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.520342    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls podName:48e7128a-0b83-44df-957e-34e25e338ec0 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:38.520320941 +0000 UTC m=+256.907668368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sv6kl" (UID: "48e7128a-0b83-44df-957e-34e25e338ec0") : secret "samples-operator-tls" not found
Jul 17 13:49:37.621127 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.621072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:37.621212 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.621196    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 13:49:37.621253 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.621213    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd9c4b87-9dsn6: secret "image-registry-tls" not found
Jul 17 13:49:37.621253 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:37.621252    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls podName:f8d6ef3b-0c52-4586-a224-7943c1b53190 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:38.621239537 +0000 UTC m=+257.008586955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls") pod "image-registry-7cd9c4b87-9dsn6" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190") : secret "image-registry-tls" not found
Jul 17 13:49:37.759696 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.759662    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m" event={"ID":"d6a6e949-2fd9-49fe-9d3d-7adc6e516721","Type":"ContainerStarted","Data":"5144dd65cdca8ef27db0c47f97a466b28e4e6778abcc8f3e1e573e5735557ca0"}
Jul 17 13:49:37.761055 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.761033    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn" event={"ID":"9a868e7f-3f04-469f-8fe4-551bda323cf1","Type":"ContainerStarted","Data":"eee8bfc421e602723dfd12fae69db0c373cfc5a8e97842d198334d40b9262b18"}
Jul 17 13:49:37.762406 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:37.762379    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" event={"ID":"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0","Type":"ContainerStarted","Data":"a17a216952152f8456ef03d29ee3304f78d0f9c66ab2e19ba2ad44b0dff1026d"}
Jul 17 13:49:38.428352 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:38.428313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:38.428539 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:38.428474    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:38.428607 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:38.428553    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls podName:ba0000c9-84a7-4f61-87e0-69d14de53c10 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:40.428531108 +0000 UTC m=+258.815878527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7r6jb" (UID: "ba0000c9-84a7-4f61-87e0-69d14de53c10") : secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:38.528819 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:38.528782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:38.528972 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:38.528902    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 13:49:38.529033 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:38.528977    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls podName:48e7128a-0b83-44df-957e-34e25e338ec0 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:40.528956633 +0000 UTC m=+258.916304054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sv6kl" (UID: "48e7128a-0b83-44df-957e-34e25e338ec0") : secret "samples-operator-tls" not found
Jul 17 13:49:38.629520 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:38.629356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:38.629520 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:38.629513    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 13:49:38.629697 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:38.629535    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd9c4b87-9dsn6: secret "image-registry-tls" not found
Jul 17 13:49:38.629697 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:38.629594    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls podName:f8d6ef3b-0c52-4586-a224-7943c1b53190 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:40.629574703 +0000 UTC m=+259.016922131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls") pod "image-registry-7cd9c4b87-9dsn6" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190") : secret "image-registry-tls" not found
Jul 17 13:49:38.766640 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:38.766608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn" event={"ID":"9a868e7f-3f04-469f-8fe4-551bda323cf1","Type":"ContainerStarted","Data":"b26862e3d6039c1d0ce3cc33da9a5943ddccbb71895abf4db47b49dc7125a5d7"}
Jul 17 13:49:38.788357 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:38.788310    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-spqnn" podStartSLOduration=1.5538170089999999 podStartE2EDuration="2.788294652s" totalImagesPullingTime="1.234477643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:49:38.787254827 +0000 UTC m=+257.174602267" watchObservedRunningTime="2026-07-17 13:49:38.788294652 +0000 UTC m=+257.175642134"
Jul 17 13:49:40.443887 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.443858    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:40.444313 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:40.443972    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:40.444313 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:40.444022    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls podName:ba0000c9-84a7-4f61-87e0-69d14de53c10 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:44.444008428 +0000 UTC m=+262.831355850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7r6jb" (UID: "ba0000c9-84a7-4f61-87e0-69d14de53c10") : secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:40.544854 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.544821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:40.544972 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:40.544921    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 13:49:40.544972 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:40.544963    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls podName:48e7128a-0b83-44df-957e-34e25e338ec0 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:44.544952213 +0000 UTC m=+262.932299635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sv6kl" (UID: "48e7128a-0b83-44df-957e-34e25e338ec0") : secret "samples-operator-tls" not found
Jul 17 13:49:40.645842 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.645818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:40.645963 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:40.645899    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 13:49:40.645963 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:40.645909    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd9c4b87-9dsn6: secret "image-registry-tls" not found
Jul 17 13:49:40.645963 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:40.645947    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls podName:f8d6ef3b-0c52-4586-a224-7943c1b53190 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:44.645934606 +0000 UTC m=+263.033282024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls") pod "image-registry-7cd9c4b87-9dsn6" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190") : secret "image-registry-tls" not found
Jul 17 13:49:40.773287 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.773244    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m" event={"ID":"d6a6e949-2fd9-49fe-9d3d-7adc6e516721","Type":"ContainerStarted","Data":"3da02ae312860d070c8281c3ce09de71f9ae6653b7dd91904b784c652180e5e5"}
Jul 17 13:49:40.774838 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.774821    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/0.log"
Jul 17 13:49:40.774932 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.774854    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0" containerID="48808c12e84bfad23d853687f235bc44bd67d738a1c5bbfb90d876f0347f8064" exitCode=255
Jul 17 13:49:40.774932 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.774875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" event={"ID":"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0","Type":"ContainerDied","Data":"48808c12e84bfad23d853687f235bc44bd67d738a1c5bbfb90d876f0347f8064"}
Jul 17 13:49:40.775059 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.775045    2575 scope.go:117] "RemoveContainer" containerID="48808c12e84bfad23d853687f235bc44bd67d738a1c5bbfb90d876f0347f8064"
Jul 17 13:49:40.846644 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:40.846598    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m" podStartSLOduration=2.470160787 podStartE2EDuration="4.846583583s" totalImagesPullingTime="2.376422796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:49:40.800185488 +0000 UTC m=+259.187532939" watchObservedRunningTime="2026-07-17 13:49:40.846583583 +0000 UTC m=+259.233931023"
Jul 17 13:49:41.440199 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.440171    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7nrbn"]
Jul 17 13:49:41.442347 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.442330    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.450732 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.450714    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 13:49:41.451057 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.450842    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 13:49:41.451057 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.450937    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 13:49:41.453374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.453359    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kldnd\""
Jul 17 13:49:41.453437 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.453360    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 13:49:41.474093 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.474071    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7nrbn"]
Jul 17 13:49:41.552013 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.551988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e581b32b-7c8b-4268-a396-e910637ed696-crio-socket\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.552107 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.552036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.552159 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.552112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.552159 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.552137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smqw\" (UniqueName: \"kubernetes.io/projected/e581b32b-7c8b-4268-a396-e910637ed696-kube-api-access-7smqw\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.552237 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.552165    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e581b32b-7c8b-4268-a396-e910637ed696-data-volume\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652572 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.652545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e581b32b-7c8b-4268-a396-e910637ed696-data-volume\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652657 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.652600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e581b32b-7c8b-4268-a396-e910637ed696-crio-socket\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.652663    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e581b32b-7c8b-4268-a396-e910637ed696-crio-socket\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652758 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.652716    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652802 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.652764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652802 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.652781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smqw\" (UniqueName: \"kubernetes.io/projected/e581b32b-7c8b-4268-a396-e910637ed696-kube-api-access-7smqw\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652896 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:41.652828    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 13:49:41.652896 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.652834    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e581b32b-7c8b-4268-a396-e910637ed696-data-volume\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.652896 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:41.652868    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:41.652896 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:41.652884    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:42.152868041 +0000 UTC m=+260.540215494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : configmap "kube-rbac-proxy" not found
Jul 17 13:49:41.653044 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:41.652927    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:42.15291028 +0000 UTC m=+260.540257733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:41.670142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.670116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smqw\" (UniqueName: \"kubernetes.io/projected/e581b32b-7c8b-4268-a396-e910637ed696-kube-api-access-7smqw\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:41.778993 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.778974    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/1.log"
Jul 17 13:49:41.779399 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.779383    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/0.log"
Jul 17 13:49:41.779498 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.779413    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0" containerID="afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21" exitCode=255
Jul 17 13:49:41.779554 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.779494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" event={"ID":"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0","Type":"ContainerDied","Data":"afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21"}
Jul 17 13:49:41.779554 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.779539    2575 scope.go:117] "RemoveContainer" containerID="48808c12e84bfad23d853687f235bc44bd67d738a1c5bbfb90d876f0347f8064"
Jul 17 13:49:41.779757 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:41.779728    2575 scope.go:117] "RemoveContainer" containerID="afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21"
Jul 17 13:49:41.779910 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:41.779879    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wmcnx_openshift-console-operator(e8b7a27d-388f-4596-ad2b-bf20ad40b2a0)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podUID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0"
Jul 17 13:49:42.156349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.156266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:42.156478 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:42.156378    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:42.156478 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.156431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:42.156478 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:42.156442    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:43.156424661 +0000 UTC m=+261.543772093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:42.156656 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:42.156539    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 13:49:42.156656 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:42.156604    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:43.156588682 +0000 UTC m=+261.543936106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : configmap "kube-rbac-proxy" not found
Jul 17 13:49:42.776876 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.776841    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"]
Jul 17 13:49:42.779483 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.779467    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"
Jul 17 13:49:42.781857 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.781835    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 13:49:42.781994 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.781972    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d8kfs\""
Jul 17 13:49:42.782077 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.781982    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 13:49:42.783883 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.783864    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/1.log"
Jul 17 13:49:42.784139 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.784126    2575 scope.go:117] "RemoveContainer" containerID="afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21"
Jul 17 13:49:42.784315 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:42.784272    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wmcnx_openshift-console-operator(e8b7a27d-388f-4596-ad2b-bf20ad40b2a0)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podUID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0"
Jul 17 13:49:42.791021 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.791001    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"]
Jul 17 13:49:42.861879 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.861854    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhdn\" (UniqueName: \"kubernetes.io/projected/ef952a93-80dd-4ca5-9db3-61f802e93105-kube-api-access-rkhdn\") pod \"migrator-7cc9c7ffd7-626wr\" (UID: \"ef952a93-80dd-4ca5-9db3-61f802e93105\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"
Jul 17 13:49:42.962353 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.962330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhdn\" (UniqueName: \"kubernetes.io/projected/ef952a93-80dd-4ca5-9db3-61f802e93105-kube-api-access-rkhdn\") pod \"migrator-7cc9c7ffd7-626wr\" (UID: \"ef952a93-80dd-4ca5-9db3-61f802e93105\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"
Jul 17 13:49:42.970135 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:42.970108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhdn\" (UniqueName: \"kubernetes.io/projected/ef952a93-80dd-4ca5-9db3-61f802e93105-kube-api-access-rkhdn\") pod \"migrator-7cc9c7ffd7-626wr\" (UID: \"ef952a93-80dd-4ca5-9db3-61f802e93105\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"
Jul 17 13:49:43.088317 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:43.088246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"
Jul 17 13:49:43.149253 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:43.149228    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58ggv_ef9b353f-aeb1-4d0b-adfb-01e472913c4b/dns-node-resolver/0.log"
Jul 17 13:49:43.164380 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:43.164353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:43.164485 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:43.164403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:43.164552 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:43.164503    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 13:49:43.164552 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:43.164514    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:43.164652 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:43.164554    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:45.164538231 +0000 UTC m=+263.551885651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : configmap "kube-rbac-proxy" not found
Jul 17 13:49:43.164652 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:43.164566    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:45.164561281 +0000 UTC m=+263.551908698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:43.204985 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:43.204961    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr"]
Jul 17 13:49:43.207901 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:49:43.207876    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef952a93_80dd_4ca5_9db3_61f802e93105.slice/crio-979c20380675b6ce95f65091bd2636f4cb31ccd5e8cd27d58f9210e66ddc28cf WatchSource:0}: Error finding container 979c20380675b6ce95f65091bd2636f4cb31ccd5e8cd27d58f9210e66ddc28cf: Status 404 returned error can't find the container with id 979c20380675b6ce95f65091bd2636f4cb31ccd5e8cd27d58f9210e66ddc28cf
Jul 17 13:49:43.787844 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:43.787812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr" event={"ID":"ef952a93-80dd-4ca5-9db3-61f802e93105","Type":"ContainerStarted","Data":"979c20380675b6ce95f65091bd2636f4cb31ccd5e8cd27d58f9210e66ddc28cf"}
Jul 17 13:49:44.348502 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:44.348485    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gvl6k_c74d1625-9ecb-45cd-bc27-3f87ed8d40b3/node-ca/0.log"
Jul 17 13:49:44.475430 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:44.475397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:44.475559 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:44.475497    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:44.475559 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:44.475539    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls podName:ba0000c9-84a7-4f61-87e0-69d14de53c10 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:52.475526589 +0000 UTC m=+270.862874011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7r6jb" (UID: "ba0000c9-84a7-4f61-87e0-69d14de53c10") : secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:44.576385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:44.576361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:44.576518 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:44.576504    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 13:49:44.576574 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:44.576554    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls podName:48e7128a-0b83-44df-957e-34e25e338ec0 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:52.576538675 +0000 UTC m=+270.963886102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sv6kl" (UID: "48e7128a-0b83-44df-957e-34e25e338ec0") : secret "samples-operator-tls" not found
Jul 17 13:49:44.677270 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:44.677207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:44.677374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:44.677324    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 13:49:44.677374 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:44.677339    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd9c4b87-9dsn6: secret "image-registry-tls" not found
Jul 17 13:49:44.677454 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:44.677382    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls podName:f8d6ef3b-0c52-4586-a224-7943c1b53190 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:52.677368226 +0000 UTC m=+271.064715647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls") pod "image-registry-7cd9c4b87-9dsn6" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190") : secret "image-registry-tls" not found
Jul 17 13:49:44.794616 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:44.794588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr" event={"ID":"ef952a93-80dd-4ca5-9db3-61f802e93105","Type":"ContainerStarted","Data":"dfd0b5d4a68ad11907c0f103183575e764cca66de0d80321de1279a152608de0"}
Jul 17 13:49:44.794924 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:44.794640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr" event={"ID":"ef952a93-80dd-4ca5-9db3-61f802e93105","Type":"ContainerStarted","Data":"24340fea4fc16b53ad7d7ad56d1cb15155ff682f78f8fff6a33faf15e7a1a6c1"}
Jul 17 13:49:44.810764 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:44.810714    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-626wr" podStartSLOduration=1.682853404 podStartE2EDuration="2.810698493s" totalImagesPullingTime="1.127845089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:49:44.810693341 +0000 UTC m=+263.198040803" watchObservedRunningTime="2026-07-17 13:49:44.810698493 +0000 UTC m=+263.198045931"
Jul 17 13:49:45.150093 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:45.150048    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g7wdl_0e1a49fa-42d8-4f65-a8a2-7497d017e06d/serve-healthcheck-canary/0.log"
Jul 17 13:49:45.181516 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:45.181486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:45.181627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:45.181554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:45.181699 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:45.181654    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 13:49:45.181699 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:45.181675    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:45.181797 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:45.181719    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:49.18170227 +0000 UTC m=+267.569049691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : configmap "kube-rbac-proxy" not found
Jul 17 13:49:45.181797 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:45.181740    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:49.181729821 +0000 UTC m=+267.569077243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:47.161565 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:47.161538    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:47.161944 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:47.161600    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:49:47.161944 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:47.161886    2575 scope.go:117] "RemoveContainer" containerID="afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21"
Jul 17 13:49:47.162043 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:47.162023    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wmcnx_openshift-console-operator(e8b7a27d-388f-4596-ad2b-bf20ad40b2a0)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podUID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0"
Jul 17 13:49:47.802949 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:47.802915    2575 scope.go:117] "RemoveContainer" containerID="afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21"
Jul 17 13:49:47.803141 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:47.803093    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wmcnx_openshift-console-operator(e8b7a27d-388f-4596-ad2b-bf20ad40b2a0)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podUID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0"
Jul 17 13:49:49.213184 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:49.213152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:49.213637 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:49.213211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:49.213637 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:49.213362    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:49.213637 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:49.213420    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:57.213403 +0000 UTC m=+275.600750423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : secret "insights-runtime-extractor-tls" not found
Jul 17 13:49:49.213637 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:49.213485    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 13:49:49.213637 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:49.213544    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm podName:e581b32b-7c8b-4268-a396-e910637ed696 nodeName:}" failed. No retries permitted until 2026-07-17 13:49:57.213529342 +0000 UTC m=+275.600876765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nrbn" (UID: "e581b32b-7c8b-4268-a396-e910637ed696") : configmap "kube-rbac-proxy" not found
Jul 17 13:49:52.539162 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.539121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:49:52.539708 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:52.539234    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:52.539708 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:49:52.539310    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls podName:ba0000c9-84a7-4f61-87e0-69d14de53c10 nodeName:}" failed. No retries permitted until 2026-07-17 13:50:08.539293678 +0000 UTC m=+286.926641096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7r6jb" (UID: "ba0000c9-84a7-4f61-87e0-69d14de53c10") : secret "cluster-monitoring-operator-tls" not found
Jul 17 13:49:52.639656 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.639617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:52.642338 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.642306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48e7128a-0b83-44df-957e-34e25e338ec0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sv6kl\" (UID: \"48e7128a-0b83-44df-957e-34e25e338ec0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:52.734860 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.734832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"
Jul 17 13:49:52.741025 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.740988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:52.743540 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.743512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"image-registry-7cd9c4b87-9dsn6\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") " pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:52.850354 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.850327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:52.860054 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.860031    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl"]
Jul 17 13:49:52.974835 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:52.974807    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd9c4b87-9dsn6"]
Jul 17 13:49:52.977368 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:49:52.977338    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d6ef3b_0c52_4586_a224_7943c1b53190.slice/crio-e498ce456173ba4dc4b1bf3586f2ef9907b511a94fdc4d042fe8b9df04616445 WatchSource:0}: Error finding container e498ce456173ba4dc4b1bf3586f2ef9907b511a94fdc4d042fe8b9df04616445: Status 404 returned error can't find the container with id e498ce456173ba4dc4b1bf3586f2ef9907b511a94fdc4d042fe8b9df04616445
Jul 17 13:49:53.823550 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:53.823313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6" event={"ID":"f8d6ef3b-0c52-4586-a224-7943c1b53190","Type":"ContainerStarted","Data":"1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1"}
Jul 17 13:49:53.823550 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:53.823383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6" event={"ID":"f8d6ef3b-0c52-4586-a224-7943c1b53190","Type":"ContainerStarted","Data":"e498ce456173ba4dc4b1bf3586f2ef9907b511a94fdc4d042fe8b9df04616445"}
Jul 17 13:49:53.823550 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:53.823426    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:49:53.824605 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:53.824578    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl" event={"ID":"48e7128a-0b83-44df-957e-34e25e338ec0","Type":"ContainerStarted","Data":"255ee6d80d93ebc24b2c52c2b1d5b6e5c5c328c799138f3f55b43716b2cb3856"}
Jul 17 13:49:53.847029 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:53.846989    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6" podStartSLOduration=17.846975063 podStartE2EDuration="17.846975063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:49:53.845259294 +0000 UTC m=+272.232606733" watchObservedRunningTime="2026-07-17 13:49:53.846975063 +0000 UTC m=+272.234322502"
Jul 17 13:49:54.829142 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:54.829105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl" event={"ID":"48e7128a-0b83-44df-957e-34e25e338ec0","Type":"ContainerStarted","Data":"bd1dbeb76e65d646ed47c4dcd97859d2220af7aef014161a6cf125d1d6528d72"}
Jul 17 13:49:54.829517 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:54.829171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl" event={"ID":"48e7128a-0b83-44df-957e-34e25e338ec0","Type":"ContainerStarted","Data":"2551156d76438177e082b8395180b98cf813ae22de2b1da84df6a8b732b72ce2"}
Jul 17 13:49:54.846085 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:54.846045    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sv6kl" podStartSLOduration=17.293712002 podStartE2EDuration="18.8460317s" totalImagesPullingTime="1.552319698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:49:54.844847126 +0000 UTC m=+273.232194576" watchObservedRunningTime="2026-07-17 13:49:54.8460317 +0000 UTC m=+273.233379164"
Jul 17 13:49:57.275447 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:57.275414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:57.275848 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:57.275600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:57.276123 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:57.276102    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e581b32b-7c8b-4268-a396-e910637ed696-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:57.278011 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:57.277992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e581b32b-7c8b-4268-a396-e910637ed696-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nrbn\" (UID: \"e581b32b-7c8b-4268-a396-e910637ed696\") " pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:57.350769 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:57.350739    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7nrbn"
Jul 17 13:49:57.480038 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:57.480013    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7nrbn"]
Jul 17 13:49:57.483149 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:49:57.483119    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode581b32b_7c8b_4268_a396_e910637ed696.slice/crio-9d0994ada372c5f71186e91807331e3a7e21a14343d8e700b47fccf0084183c7 WatchSource:0}: Error finding container 9d0994ada372c5f71186e91807331e3a7e21a14343d8e700b47fccf0084183c7: Status 404 returned error can't find the container with id 9d0994ada372c5f71186e91807331e3a7e21a14343d8e700b47fccf0084183c7
Jul 17 13:49:57.841598 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:49:57.841562    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nrbn" event={"ID":"e581b32b-7c8b-4268-a396-e910637ed696","Type":"ContainerStarted","Data":"9d0994ada372c5f71186e91807331e3a7e21a14343d8e700b47fccf0084183c7"}
Jul 17 13:50:02.070692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.070661    2575 scope.go:117] "RemoveContainer" containerID="afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21"
Jul 17 13:50:02.533929 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:50:02.533892    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5x48l" podUID="2d686534-59bf-492c-89e5-98689ba395df"
Jul 17 13:50:02.865016 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.864866    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 13:50:02.865412 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.865392    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/1.log"
Jul 17 13:50:02.865504 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.865435    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0" containerID="a118c6aef066a217b5810f70646dd0baccead979cbea53318e2136713f51438f" exitCode=255
Jul 17 13:50:02.865566 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.865518    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" event={"ID":"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0","Type":"ContainerDied","Data":"a118c6aef066a217b5810f70646dd0baccead979cbea53318e2136713f51438f"}
Jul 17 13:50:02.865716 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.865624    2575 scope.go:117] "RemoveContainer" containerID="afebd852ffdf10d2d1d6924b82cd282bc8827c1d1a486dfd8bc9aaefad68de21"
Jul 17 13:50:02.866033 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.866017    2575 scope.go:117] "RemoveContainer" containerID="a118c6aef066a217b5810f70646dd0baccead979cbea53318e2136713f51438f"
Jul 17 13:50:02.866377 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:50:02.866353    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-wmcnx_openshift-console-operator(e8b7a27d-388f-4596-ad2b-bf20ad40b2a0)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podUID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0"
Jul 17 13:50:02.868127 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.868103    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nrbn" event={"ID":"e581b32b-7c8b-4268-a396-e910637ed696","Type":"ContainerStarted","Data":"3d6526a4c063c19dfd824c59363f97b785de375f0ced78bff313658a90209125"}
Jul 17 13:50:02.868242 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:02.868142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x48l"
Jul 17 13:50:03.872938 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:03.872907    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 13:50:05.882235 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:05.882201    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nrbn" event={"ID":"e581b32b-7c8b-4268-a396-e910637ed696","Type":"ContainerStarted","Data":"3ff7cc08c4c65f48256bb5b95c73058aeb9fc97f87613c2ef4294e453adb6862"}
Jul 17 13:50:06.447081 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:06.447045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:50:06.449513 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:06.449488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d686534-59bf-492c-89e5-98689ba395df-metrics-tls\") pod \"dns-default-5x48l\" (UID: \"2d686534-59bf-492c-89e5-98689ba395df\") " pod="openshift-dns/dns-default-5x48l"
Jul 17 13:50:06.471844 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:06.471812    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gl8vw\""
Jul 17 13:50:06.479746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:06.479722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x48l"
Jul 17 13:50:06.639641 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:06.639614    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5x48l"]
Jul 17 13:50:06.642649 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:50:06.642617    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d686534_59bf_492c_89e5_98689ba395df.slice/crio-a1360a453d96dca6ba66708c8ac02e3fe801b6383cc978a73c74a1982ee63bfc WatchSource:0}: Error finding container a1360a453d96dca6ba66708c8ac02e3fe801b6383cc978a73c74a1982ee63bfc: Status 404 returned error can't find the container with id a1360a453d96dca6ba66708c8ac02e3fe801b6383cc978a73c74a1982ee63bfc
Jul 17 13:50:06.887043 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:06.887009    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x48l" event={"ID":"2d686534-59bf-492c-89e5-98689ba395df","Type":"ContainerStarted","Data":"a1360a453d96dca6ba66708c8ac02e3fe801b6383cc978a73c74a1982ee63bfc"}
Jul 17 13:50:06.957609 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:06.957580    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd9c4b87-9dsn6"]
Jul 17 13:50:07.162216 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:07.162141    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:50:07.162216 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:07.162205    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:50:07.162725 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:07.162592    2575 scope.go:117] "RemoveContainer" containerID="a118c6aef066a217b5810f70646dd0baccead979cbea53318e2136713f51438f"
Jul 17 13:50:07.162837 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:50:07.162782    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-wmcnx_openshift-console-operator(e8b7a27d-388f-4596-ad2b-bf20ad40b2a0)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podUID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0"
Jul 17 13:50:08.570250 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:08.570211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:50:08.572685 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:08.572659    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba0000c9-84a7-4f61-87e0-69d14de53c10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7r6jb\" (UID: \"ba0000c9-84a7-4f61-87e0-69d14de53c10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:50:08.842733 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:08.842648    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"
Jul 17 13:50:09.307786 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.307750    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb"]
Jul 17 13:50:09.311262 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:50:09.311239    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0000c9_84a7_4f61_87e0_69d14de53c10.slice/crio-288d9303859c7419e225b7179243773f8711deec715e087950fd3de1a014b3c5 WatchSource:0}: Error finding container 288d9303859c7419e225b7179243773f8711deec715e087950fd3de1a014b3c5: Status 404 returned error can't find the container with id 288d9303859c7419e225b7179243773f8711deec715e087950fd3de1a014b3c5
Jul 17 13:50:09.899397 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.899354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x48l" event={"ID":"2d686534-59bf-492c-89e5-98689ba395df","Type":"ContainerStarted","Data":"2d7201aa6b86a4be8f8c7839c0a99845e8c276947efe6b16cb0c1f997057c54f"}
Jul 17 13:50:09.899848 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.899423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x48l" event={"ID":"2d686534-59bf-492c-89e5-98689ba395df","Type":"ContainerStarted","Data":"b025124e3fcc8b02257aae116ba8023b8a324ddb03c8ed0b60fefc39b6c53bc6"}
Jul 17 13:50:09.899848 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.899492    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5x48l"
Jul 17 13:50:09.900899 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.900867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb" event={"ID":"ba0000c9-84a7-4f61-87e0-69d14de53c10","Type":"ContainerStarted","Data":"288d9303859c7419e225b7179243773f8711deec715e087950fd3de1a014b3c5"}
Jul 17 13:50:09.903410 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.903383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nrbn" event={"ID":"e581b32b-7c8b-4268-a396-e910637ed696","Type":"ContainerStarted","Data":"969888a2c81b7619d7f843a9d570c3cd7cd05364edd1c96fcd59385800eaf425"}
Jul 17 13:50:09.918059 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.918001    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5x48l" podStartSLOduration=251.381625274 podStartE2EDuration="4m13.9179815s" totalImagesPullingTime="2.536356226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:09.917202606 +0000 UTC m=+288.304550047" watchObservedRunningTime="2026-07-17 13:50:09.9179815 +0000 UTC m=+288.305328940"
Jul 17 13:50:09.935204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:09.935166    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7nrbn" podStartSLOduration=17.341902698 podStartE2EDuration="28.935148827s" totalImagesPullingTime="11.593246129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:49:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:09.935024267 +0000 UTC m=+288.322371707" watchObservedRunningTime="2026-07-17 13:50:09.935148827 +0000 UTC m=+288.322496266"
Jul 17 13:50:11.911793 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:11.911761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb" event={"ID":"ba0000c9-84a7-4f61-87e0-69d14de53c10","Type":"ContainerStarted","Data":"f7d1cb42c555fee90dfe5dcb84dd2c8e0becce53f5e369f98446d34adac3a31b"}
Jul 17 13:50:11.928782 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:11.928746    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb" podStartSLOduration=33.977808115 podStartE2EDuration="35.928730683s" totalImagesPullingTime="1.950922568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:11.927734126 +0000 UTC m=+290.315081565" watchObservedRunningTime="2026-07-17 13:50:11.928730683 +0000 UTC m=+290.316078122"
Jul 17 13:50:14.815079 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.815040    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"]
Jul 17 13:50:14.817987 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.817970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:14.822068 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.822042    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 17 13:50:14.822068 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.822054    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 17 13:50:14.822288 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.822077    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4nt6l\""
Jul 17 13:50:14.822288 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.822064    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 13:50:14.832066 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.832043    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"]
Jul 17 13:50:14.911542 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.911514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e300a68-c944-416c-9d7e-55ab8f3538c3-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:14.911659 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.911553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e300a68-c944-416c-9d7e-55ab8f3538c3-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:14.911659 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.911573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e300a68-c944-416c-9d7e-55ab8f3538c3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:14.911729 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:14.911657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwvp\" (UniqueName: \"kubernetes.io/projected/0e300a68-c944-416c-9d7e-55ab8f3538c3-kube-api-access-ggwvp\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.012573 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.012549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e300a68-c944-416c-9d7e-55ab8f3538c3-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.012695 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.012577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e300a68-c944-416c-9d7e-55ab8f3538c3-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.012695 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.012596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e300a68-c944-416c-9d7e-55ab8f3538c3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.012695 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.012630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwvp\" (UniqueName: \"kubernetes.io/projected/0e300a68-c944-416c-9d7e-55ab8f3538c3-kube-api-access-ggwvp\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.013183 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.013164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e300a68-c944-416c-9d7e-55ab8f3538c3-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.015056 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.015038    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e300a68-c944-416c-9d7e-55ab8f3538c3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.015126 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.015109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e300a68-c944-416c-9d7e-55ab8f3538c3-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.020930 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.020907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwvp\" (UniqueName: \"kubernetes.io/projected/0e300a68-c944-416c-9d7e-55ab8f3538c3-kube-api-access-ggwvp\") pod \"prometheus-operator-5945b5cfd-zdgts\" (UID: \"0e300a68-c944-416c-9d7e-55ab8f3538c3\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.127392 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.127343    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"
Jul 17 13:50:15.243018 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.242994    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zdgts"]
Jul 17 13:50:15.245533 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:50:15.245505    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e300a68_c944_416c_9d7e_55ab8f3538c3.slice/crio-2fc8c48044df72cebc1faf7fe973efbf17131e5fe63decce3fd7f7e81195139e WatchSource:0}: Error finding container 2fc8c48044df72cebc1faf7fe973efbf17131e5fe63decce3fd7f7e81195139e: Status 404 returned error can't find the container with id 2fc8c48044df72cebc1faf7fe973efbf17131e5fe63decce3fd7f7e81195139e
Jul 17 13:50:15.927225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:15.927188    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts" event={"ID":"0e300a68-c944-416c-9d7e-55ab8f3538c3","Type":"ContainerStarted","Data":"2fc8c48044df72cebc1faf7fe973efbf17131e5fe63decce3fd7f7e81195139e"}
Jul 17 13:50:16.932215 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:16.932180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts" event={"ID":"0e300a68-c944-416c-9d7e-55ab8f3538c3","Type":"ContainerStarted","Data":"e3283997c0ec3c6b2ddf8e23ac322b0e57281687b55a87a4f0a3c7f5595d3b84"}
Jul 17 13:50:16.932603 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:16.932234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts" event={"ID":"0e300a68-c944-416c-9d7e-55ab8f3538c3","Type":"ContainerStarted","Data":"97d2626e804e1d5ce76ee417d1e3cd99b7505186c98a6de196ae923281186678"}
Jul 17 13:50:16.950124 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:16.950037    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zdgts" podStartSLOduration=1.860032179 podStartE2EDuration="2.950017781s" totalImagesPullingTime="1.089985602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:16.949158001 +0000 UTC m=+295.336505443" watchObservedRunningTime="2026-07-17 13:50:16.950017781 +0000 UTC m=+295.337365222"
Jul 17 13:50:16.963108 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:16.963083    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:50:18.939661 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:18.939640    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 13:50:18.939929 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:18.939676    2575 generic.go:356] "Generic (PLEG): container finished" podID="ba0000c9-84a7-4f61-87e0-69d14de53c10" containerID="f7d1cb42c555fee90dfe5dcb84dd2c8e0becce53f5e369f98446d34adac3a31b" exitCode=2
Jul 17 13:50:18.939929 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:18.939709    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb" event={"ID":"ba0000c9-84a7-4f61-87e0-69d14de53c10","Type":"ContainerDied","Data":"f7d1cb42c555fee90dfe5dcb84dd2c8e0becce53f5e369f98446d34adac3a31b"}
Jul 17 13:50:18.940000 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:18.939958    2575 scope.go:117] "RemoveContainer" containerID="f7d1cb42c555fee90dfe5dcb84dd2c8e0becce53f5e369f98446d34adac3a31b"
Jul 17 13:50:19.069597 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:50:19.069559    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cr2qc" podUID="8ec62e35-b2e6-4161-871f-3463c827961d"
Jul 17 13:50:19.909346 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:19.909320    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5x48l"
Jul 17 13:50:19.945465 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:19.945440    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 13:50:19.945818 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:19.945534    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7r6jb" event={"ID":"ba0000c9-84a7-4f61-87e0-69d14de53c10","Type":"ContainerStarted","Data":"b0ee30fbfa35eab4988f020336583d5142fd8ef7630f7e8022891a8f163395b8"}
Jul 17 13:50:21.944832 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:21.944807    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 13:50:21.945215 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:21.945193    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 13:50:21.948126 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:21.948106    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 13:50:21.948858 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:21.948835    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 13:50:21.955629 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:21.955610    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:50:21.956406 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:21.956385    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:50:21.960650 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:21.960625    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 13:50:22.070314 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.070293    2575 scope.go:117] "RemoveContainer" containerID="a118c6aef066a217b5810f70646dd0baccead979cbea53318e2136713f51438f"
Jul 17 13:50:22.070469 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:50:22.070443    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-wmcnx_openshift-console-operator(e8b7a27d-388f-4596-ad2b-bf20ad40b2a0)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podUID="e8b7a27d-388f-4596-ad2b-bf20ad40b2a0"
Jul 17 13:50:22.610770 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.607549    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"]
Jul 17 13:50:22.612876 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.611693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.613937 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.613920    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rxgzq\""
Jul 17 13:50:22.614384 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.614362    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 17 13:50:22.614478 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.614366    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 17 13:50:22.622721 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.622701    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"]
Jul 17 13:50:22.635780 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.635760    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n8h5w"]
Jul 17 13:50:22.638678 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.638664    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.640843 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.640823    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 13:50:22.640972 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.640954    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 13:50:22.641110 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.641083    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 13:50:22.641211 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.641194    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bbfks\""
Jul 17 13:50:22.657797 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-tls\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.657896 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6b3057c-b837-46ea-903f-f298702b6565-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.657896 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14444c67-24b3-4ae7-9e92-d20b7d012943-metrics-client-ca\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.657896 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-sys\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.657896 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pprm\" (UniqueName: \"kubernetes.io/projected/d6b3057c-b837-46ea-903f-f298702b6565-kube-api-access-8pprm\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.658055 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-root\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.658055 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.658055 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khw5l\" (UniqueName: \"kubernetes.io/projected/14444c67-24b3-4ae7-9e92-d20b7d012943-kube-api-access-khw5l\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.658055 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.657993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.658055 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.658024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3057c-b837-46ea-903f-f298702b6565-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.658315 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.658048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-textfile\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.658315 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.658126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6b3057c-b837-46ea-903f-f298702b6565-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.658315 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.658165    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-wtmp\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759349 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759475 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khw5l\" (UniqueName: \"kubernetes.io/projected/14444c67-24b3-4ae7-9e92-d20b7d012943-kube-api-access-khw5l\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759475 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759475 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759402    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3057c-b837-46ea-903f-f298702b6565-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.759475 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-textfile\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759475 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6b3057c-b837-46ea-903f-f298702b6565-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.759720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-wtmp\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-tls\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6b3057c-b837-46ea-903f-f298702b6565-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.759720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14444c67-24b3-4ae7-9e92-d20b7d012943-metrics-client-ca\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-sys\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.759720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pprm\" (UniqueName: \"kubernetes.io/projected/d6b3057c-b837-46ea-903f-f298702b6565-kube-api-access-8pprm\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.759720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-root\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.760021 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759825    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-wtmp\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.760021 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-sys\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.760021 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.759878    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14444c67-24b3-4ae7-9e92-d20b7d012943-root\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.760201 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.760158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3057c-b837-46ea-903f-f298702b6565-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.760444 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.760423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-textfile\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.760547 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.760524    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14444c67-24b3-4ae7-9e92-d20b7d012943-metrics-client-ca\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.760606 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.760531    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.762081 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.762058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.762160 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.762144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6b3057c-b837-46ea-903f-f298702b6565-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.762268 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.762249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6b3057c-b837-46ea-903f-f298702b6565-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.762358 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.762338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14444c67-24b3-4ae7-9e92-d20b7d012943-node-exporter-tls\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.767536 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.767512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pprm\" (UniqueName: \"kubernetes.io/projected/d6b3057c-b837-46ea-903f-f298702b6565-kube-api-access-8pprm\") pod \"openshift-state-metrics-79f6958cdf-6tlwh\" (UID: \"d6b3057c-b837-46ea-903f-f298702b6565\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.768338 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.768249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khw5l\" (UniqueName: \"kubernetes.io/projected/14444c67-24b3-4ae7-9e92-d20b7d012943-kube-api-access-khw5l\") pod \"node-exporter-n8h5w\" (UID: \"14444c67-24b3-4ae7-9e92-d20b7d012943\") " pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:22.921553 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.921496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"
Jul 17 13:50:22.949478 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:22.949455    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n8h5w"
Jul 17 13:50:23.053299 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:23.053246    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh"]
Jul 17 13:50:23.056048 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:50:23.056022    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b3057c_b837_46ea_903f_f298702b6565.slice/crio-716179751ce3a8c452a882b30b6e35593c89828fb28cc94f2e149672977451b0 WatchSource:0}: Error finding container 716179751ce3a8c452a882b30b6e35593c89828fb28cc94f2e149672977451b0: Status 404 returned error can't find the container with id 716179751ce3a8c452a882b30b6e35593c89828fb28cc94f2e149672977451b0
Jul 17 13:50:23.165615 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:23.165599    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:50:23.959498 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:23.959457    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8h5w" event={"ID":"14444c67-24b3-4ae7-9e92-d20b7d012943","Type":"ContainerStarted","Data":"0faccca283c82910acb636a1b84852076b590edda022cc60d9f7842e7872fc46"}
Jul 17 13:50:23.961487 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:23.961462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh" event={"ID":"d6b3057c-b837-46ea-903f-f298702b6565","Type":"ContainerStarted","Data":"dbfea7ec89ccfa99f085b6648330f58317ac25e6bf4fc80f99a3a767c22fcfdf"}
Jul 17 13:50:23.961616 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:23.961507    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh" event={"ID":"d6b3057c-b837-46ea-903f-f298702b6565","Type":"ContainerStarted","Data":"c143f60eeb9f5dbefc132814a01ebaa361b1b81aac7ece878a31e117488b8eb0"}
Jul 17 13:50:23.961616 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:23.961527    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh" event={"ID":"d6b3057c-b837-46ea-903f-f298702b6565","Type":"ContainerStarted","Data":"716179751ce3a8c452a882b30b6e35593c89828fb28cc94f2e149672977451b0"}
Jul 17 13:50:24.780655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.780617    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-579d688677-nvtnv"]
Jul 17 13:50:24.784014 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.783993    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.786413 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.786388    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 17 13:50:24.786633 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.786448    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 17 13:50:24.786701 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.786688    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 17 13:50:24.786783 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.786701    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xzsj8\""
Jul 17 13:50:24.786783 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.786705    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 17 13:50:24.786930 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.786915    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 17 13:50:24.787143 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.787130    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dbleaen4nieh9\""
Jul 17 13:50:24.799912 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.799894    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-579d688677-nvtnv"]
Jul 17 13:50:24.876491 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-grpc-tls\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.876589 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.876589 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876525    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5bb\" (UniqueName: \"kubernetes.io/projected/67811a6a-bfe3-41e5-a137-5bf62d77c126-kube-api-access-cr5bb\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.876589 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.876751 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67811a6a-bfe3-41e5-a137-5bf62d77c126-metrics-client-ca\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.876751 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.876751 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.876869 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.876764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-tls\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.966604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.966583    2575 generic.go:356] "Generic (PLEG): container finished" podID="14444c67-24b3-4ae7-9e92-d20b7d012943" containerID="3eea297f8bd3edacc4b213deb603477cd4e440f657c7e64fd77fe4dfd058209f" exitCode=0
Jul 17 13:50:24.967014 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.966653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8h5w" event={"ID":"14444c67-24b3-4ae7-9e92-d20b7d012943","Type":"ContainerDied","Data":"3eea297f8bd3edacc4b213deb603477cd4e440f657c7e64fd77fe4dfd058209f"}
Jul 17 13:50:24.968969 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.968903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh" event={"ID":"d6b3057c-b837-46ea-903f-f298702b6565","Type":"ContainerStarted","Data":"cd906d9c2b0561fd5ed1c61351be34d3c5316f14e04fa6bd92ab3dc3bad63a7d"}
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5bb\" (UniqueName: \"kubernetes.io/projected/67811a6a-bfe3-41e5-a137-5bf62d77c126-kube-api-access-cr5bb\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67811a6a-bfe3-41e5-a137-5bf62d77c126-metrics-client-ca\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-tls\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.982468 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.978759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-grpc-tls\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.983669 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.983619    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67811a6a-bfe3-41e5-a137-5bf62d77c126-metrics-client-ca\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.986555 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.986114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.986555 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.986476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-tls\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.986555 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.986546    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.986814 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.986549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.986864 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.986835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.987103 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.987082    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67811a6a-bfe3-41e5-a137-5bf62d77c126-secret-grpc-tls\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:24.991116 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:24.991080    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5bb\" (UniqueName: \"kubernetes.io/projected/67811a6a-bfe3-41e5-a137-5bf62d77c126-kube-api-access-cr5bb\") pod \"thanos-querier-579d688677-nvtnv\" (UID: \"67811a6a-bfe3-41e5-a137-5bf62d77c126\") " pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:25.003560 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:25.003521    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6tlwh" podStartSLOduration=1.9791330459999998 podStartE2EDuration="3.003509263s" totalImagesPullingTime="1.024376217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:25.003188591 +0000 UTC m=+303.390536031" watchObservedRunningTime="2026-07-17 13:50:25.003509263 +0000 UTC m=+303.390856701"
Jul 17 13:50:25.093573 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:25.093549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:25.226749 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:25.226622    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-579d688677-nvtnv"]
Jul 17 13:50:25.228989 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:50:25.228963    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67811a6a_bfe3_41e5_a137_5bf62d77c126.slice/crio-e9bdcaff44f0d7f0070b4bda3467e5ab27e0b77fff23f8978b5db46c40c9b37f WatchSource:0}: Error finding container e9bdcaff44f0d7f0070b4bda3467e5ab27e0b77fff23f8978b5db46c40c9b37f: Status 404 returned error can't find the container with id e9bdcaff44f0d7f0070b4bda3467e5ab27e0b77fff23f8978b5db46c40c9b37f
Jul 17 13:50:25.974677 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:25.974640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" event={"ID":"67811a6a-bfe3-41e5-a137-5bf62d77c126","Type":"ContainerStarted","Data":"e9bdcaff44f0d7f0070b4bda3467e5ab27e0b77fff23f8978b5db46c40c9b37f"}
Jul 17 13:50:25.977340 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:25.977312    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8h5w" event={"ID":"14444c67-24b3-4ae7-9e92-d20b7d012943","Type":"ContainerStarted","Data":"e5c3e0a4416fdcd3ecfb3010d9a729fdc36c33eeb11e5e25636d5e5171605735"}
Jul 17 13:50:25.977486 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:25.977364    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8h5w" event={"ID":"14444c67-24b3-4ae7-9e92-d20b7d012943","Type":"ContainerStarted","Data":"db24a3991b68a90a20628299920a866475bae18114bcfe4aef4c0a897c02705e"}
Jul 17 13:50:25.998744 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:25.998694    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n8h5w" podStartSLOduration=2.708895612 podStartE2EDuration="3.998677356s" totalImagesPullingTime="1.224709949s" totalInitContainerRuntime="65.071795ms" isStatefulPod=true podCreationTimestamp="2026-07-17 13:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:25.997374353 +0000 UTC m=+304.384721794" watchObservedRunningTime="2026-07-17 13:50:25.998677356 +0000 UTC m=+304.386024796"
Jul 17 13:50:27.987069 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:27.987037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" event={"ID":"67811a6a-bfe3-41e5-a137-5bf62d77c126","Type":"ContainerStarted","Data":"2d7d83d13038db6b7236acad651e351d6358f442c2253b6829e31f793017aebc"}
Jul 17 13:50:27.987544 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:27.987101    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" event={"ID":"67811a6a-bfe3-41e5-a137-5bf62d77c126","Type":"ContainerStarted","Data":"9278de9b7c79b81698967e939261f1cad6e22cc25c43c5d088fd67021f6e0859"}
Jul 17 13:50:27.987544 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:27.987127    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" event={"ID":"67811a6a-bfe3-41e5-a137-5bf62d77c126","Type":"ContainerStarted","Data":"4dd7c709686efef8eb4e8d11301d0457b134a499e3aa0863c03e626afff76773"}
Jul 17 13:50:28.899269 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.899232    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:50:28.903331 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.903307    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:28.906603 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.906580    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 13:50:28.906720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.906629    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 13:50:28.906720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.906682    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 13:50:28.906830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.906591    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 13:50:28.906830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.906760    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r5p5l\""
Jul 17 13:50:28.906996 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.906980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 13:50:28.907066 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.906982    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 13:50:28.907299 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907266    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 13:50:28.907389 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907270    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 13:50:28.907460 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907299    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 13:50:28.907520 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907311    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 13:50:28.907719 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907695    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 13:50:28.907719 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907709    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 13:50:28.907846 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907701    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-78j28bmbmr6n0\""
Jul 17 13:50:28.907846 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.907770    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 13:50:28.908060 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.908044    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 13:50:28.911561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.911541    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 13:50:28.928374 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.928350    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:50:28.993542 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.993509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" event={"ID":"67811a6a-bfe3-41e5-a137-5bf62d77c126","Type":"ContainerStarted","Data":"dfcecd0ed191b0ca12397451590e88256a8b5757ac368781c9fa6cb76204289d"}
Jul 17 13:50:28.993884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.993567    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" event={"ID":"67811a6a-bfe3-41e5-a137-5bf62d77c126","Type":"ContainerStarted","Data":"4fd82a21c1886bb993b62c9a0a3bbcc67bbcc52c6b6d635662a082156eab41dd"}
Jul 17 13:50:28.993884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.993593    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" event={"ID":"67811a6a-bfe3-41e5-a137-5bf62d77c126","Type":"ContainerStarted","Data":"a301eb9f7230e7c107dcef580505c8a2db4cb5e68445186471e961448f21fb8c"}
Jul 17 13:50:28.993884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:28.993638    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:29.016028 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.015999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016028 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016180 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016180 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016180 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016180 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016381 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016182    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016381 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016381 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016381 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016297    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016381 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-config\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016381 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016430    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016464    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96km5\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-kube-api-access-96km5\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016604 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.016821 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.016620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.044161 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.044108    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv" podStartSLOduration=2.412192387 podStartE2EDuration="5.04409541s" totalImagesPullingTime="2.631903023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:50:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:29.042466903 +0000 UTC m=+307.429814342" watchObservedRunningTime="2026-07-17 13:50:29.04409541 +0000 UTC m=+307.431442850"
Jul 17 13:50:29.117607 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.117730 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.117780 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.117829 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.117829 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.117927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.117927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.117927 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.117956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118018    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-config\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96km5\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-kube-api-access-96km5\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118497 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118497 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118497 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118497 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118497 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.118727 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.118514    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.119251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.119560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.119967    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.120563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.120814    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.121007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.121122    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.121222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.121672    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.122302 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.121723    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.123715 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.123393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.123715 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.123510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-config\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.123715 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.123670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.124160 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.124132    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.124364 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.124313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.125140 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.124938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.125140 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.125096    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.126892 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.126869    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.127746 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.127726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96km5\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-kube-api-access-96km5\") pod \"prometheus-k8s-0\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.215080 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.215051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:50:29.364216 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.364188    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:50:29.365621 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:50:29.365594    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9e3383_f589_44d5_925e_db15a0c1b454.slice/crio-8be72f8f6077bc9663016402c8e42e5bf876a13ff8568f309695dafb62884138 WatchSource:0}: Error finding container 8be72f8f6077bc9663016402c8e42e5bf876a13ff8568f309695dafb62884138: Status 404 returned error can't find the container with id 8be72f8f6077bc9663016402c8e42e5bf876a13ff8568f309695dafb62884138
Jul 17 13:50:29.998638 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:29.998601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerStarted","Data":"8be72f8f6077bc9663016402c8e42e5bf876a13ff8568f309695dafb62884138"}
Jul 17 13:50:30.069559 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:30.069531    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:50:31.003881 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:31.003843    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1" exitCode=0
Jul 17 13:50:31.004239 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:31.003929    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"}
Jul 17 13:50:31.980424 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:31.980381    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6" podUID="f8d6ef3b-0c52-4586-a224-7943c1b53190" containerName="registry" containerID="cri-o://1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1" gracePeriod=30
Jul 17 13:50:32.283375 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.283343    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:50:32.347830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.347808    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28gcw\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-kube-api-access-28gcw\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.347941 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.347840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-image-registry-private-configuration\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.347941 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.347863    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.347941 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.347910    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-trusted-ca\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.348088 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.347944    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-bound-sa-token\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.348088 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.347966    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-installation-pull-secrets\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.348088 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.348014    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-certificates\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.348088 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.348051    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d6ef3b-0c52-4586-a224-7943c1b53190-ca-trust-extracted\") pod \"f8d6ef3b-0c52-4586-a224-7943c1b53190\" (UID: \"f8d6ef3b-0c52-4586-a224-7943c1b53190\") "
Jul 17 13:50:32.348523 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.348460    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:50:32.348649 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.348523    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:50:32.350433 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.350365    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:50:32.350541 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.350501    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:50:32.351039 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.351018    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:50:32.351129 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.351081    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:50:32.351182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.351164    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-kube-api-access-28gcw" (OuterVolumeSpecName: "kube-api-access-28gcw") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "kube-api-access-28gcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:50:32.358325 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.358272    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d6ef3b-0c52-4586-a224-7943c1b53190-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f8d6ef3b-0c52-4586-a224-7943c1b53190" (UID: "f8d6ef3b-0c52-4586-a224-7943c1b53190"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:50:32.448831 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448811    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8d6ef3b-0c52-4586-a224-7943c1b53190-ca-trust-extracted\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:32.448964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448832    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gcw\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-kube-api-access-28gcw\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:32.448964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448848    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-image-registry-private-configuration\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:32.448964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448860    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:32.448964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448871    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-trusted-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:32.448964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448882    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d6ef3b-0c52-4586-a224-7943c1b53190-bound-sa-token\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:32.448964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448894    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8d6ef3b-0c52-4586-a224-7943c1b53190-installation-pull-secrets\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:32.448964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:32.448906    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8d6ef3b-0c52-4586-a224-7943c1b53190-registry-certificates\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:50:33.013849 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:33.013820    2575 generic.go:356] "Generic (PLEG): container finished" podID="f8d6ef3b-0c52-4586-a224-7943c1b53190" containerID="1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1" exitCode=0
Jul 17 13:50:33.014012 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:33.013876    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6"
Jul 17 13:50:33.014012 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:33.013913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6" event={"ID":"f8d6ef3b-0c52-4586-a224-7943c1b53190","Type":"ContainerDied","Data":"1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1"}
Jul 17 13:50:33.014012 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:33.013974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd9c4b87-9dsn6" event={"ID":"f8d6ef3b-0c52-4586-a224-7943c1b53190","Type":"ContainerDied","Data":"e498ce456173ba4dc4b1bf3586f2ef9907b511a94fdc4d042fe8b9df04616445"}
Jul 17 13:50:33.014012 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:33.014004    2575 scope.go:117] "RemoveContainer" containerID="1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1"
Jul 17 13:50:33.037115 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:33.037090    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd9c4b87-9dsn6"]
Jul 17 13:50:33.046292 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:33.046257    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cd9c4b87-9dsn6"]
Jul 17 13:50:34.076051 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:34.075717    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d6ef3b-0c52-4586-a224-7943c1b53190" path="/var/lib/kubelet/pods/f8d6ef3b-0c52-4586-a224-7943c1b53190/volumes"
Jul 17 13:50:34.205028 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:34.204871    2575 scope.go:117] "RemoveContainer" containerID="1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1"
Jul 17 13:50:34.205201 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:50:34.205177    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1\": container with ID starting with 1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1 not found: ID does not exist" containerID="1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1"
Jul 17 13:50:34.205294 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:34.205212    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1"} err="failed to get container status \"1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1\": rpc error: code = NotFound desc = could not find container \"1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1\": container with ID starting with 1ca2c48d5d7cdf693a336093e538a27cd954506b10abac98d543417081dd29d1 not found: ID does not exist"
Jul 17 13:50:35.004764 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.004739    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-579d688677-nvtnv"
Jul 17 13:50:35.026806 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.026780    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerStarted","Data":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"}
Jul 17 13:50:35.026935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.026833    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerStarted","Data":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"}
Jul 17 13:50:35.026935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.026858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerStarted","Data":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"}
Jul 17 13:50:35.026935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.026883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerStarted","Data":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"}
Jul 17 13:50:35.026935 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.026910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerStarted","Data":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"}
Jul 17 13:50:35.027057 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.026938    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerStarted","Data":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"}
Jul 17 13:50:35.058112 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:35.058030    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.862656463 podStartE2EDuration="7.058010444s" totalImagesPullingTime="4.174755579s" totalInitContainerRuntime="20.598402ms" isStatefulPod=false podCreationTimestamp="2026-07-17 13:50:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:35.056586089 +0000 UTC m=+313.443933542" watchObservedRunningTime="2026-07-17 13:50:35.058010444 +0000 UTC m=+313.445357901"
Jul 17 13:50:36.069830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:36.069804    2575 scope.go:117] "RemoveContainer" containerID="a118c6aef066a217b5810f70646dd0baccead979cbea53318e2136713f51438f"
Jul 17 13:50:37.035149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:37.035118    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 13:50:37.035352 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:37.035176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" event={"ID":"e8b7a27d-388f-4596-ad2b-bf20ad40b2a0","Type":"ContainerStarted","Data":"8991453c71727a8807248a25a1a631e78c47a3541d2154e1e3685cd89486f260"}
Jul 17 13:50:37.035450 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:37.035431    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:50:37.039944 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:37.039921    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx"
Jul 17 13:50:37.056693 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:37.056652    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-wmcnx" podStartSLOduration=58.601854048 podStartE2EDuration="1m1.056633485s" totalImagesPullingTime="2.454779437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:50:37.055750562 +0000 UTC m=+315.443098013" watchObservedRunningTime="2026-07-17 13:50:37.056633485 +0000 UTC m=+315.443980978"
Jul 17 13:50:39.216332 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:50:39.216301    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:01.120338 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:01.120304    2575 generic.go:356] "Generic (PLEG): container finished" podID="d6a6e949-2fd9-49fe-9d3d-7adc6e516721" containerID="3da02ae312860d070c8281c3ce09de71f9ae6653b7dd91904b784c652180e5e5" exitCode=0
Jul 17 13:51:01.120338 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:01.120341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m" event={"ID":"d6a6e949-2fd9-49fe-9d3d-7adc6e516721","Type":"ContainerDied","Data":"3da02ae312860d070c8281c3ce09de71f9ae6653b7dd91904b784c652180e5e5"}
Jul 17 13:51:01.120771 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:01.120611    2575 scope.go:117] "RemoveContainer" containerID="3da02ae312860d070c8281c3ce09de71f9ae6653b7dd91904b784c652180e5e5"
Jul 17 13:51:02.125219 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:02.125188    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n5k7m" event={"ID":"d6a6e949-2fd9-49fe-9d3d-7adc6e516721","Type":"ContainerStarted","Data":"a1728baf87a19f1baabb2250e8fa06c1af1cc5e4fbc7a866e2dee2260b7461fb"}
Jul 17 13:51:29.216005 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:29.215959    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:29.236806 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:29.236772    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:29.252096 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:29.252069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:36.035411 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:36.035365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:51:36.037904 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:36.037881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec62e35-b2e6-4161-871f-3463c827961d-metrics-certs\") pod \"network-metrics-daemon-cr2qc\" (UID: \"8ec62e35-b2e6-4161-871f-3463c827961d\") " pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:51:36.072454 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:36.072424    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9xnsb\""
Jul 17 13:51:36.080567 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:36.080540    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cr2qc"
Jul 17 13:51:36.205640 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:36.205611    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cr2qc"]
Jul 17 13:51:36.208907 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:51:36.208879    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec62e35_b2e6_4161_871f_3463c827961d.slice/crio-26d89c0c0515f96c224e3b46815c7aacc546025c37ea61c18fc3feeec2bff45c WatchSource:0}: Error finding container 26d89c0c0515f96c224e3b46815c7aacc546025c37ea61c18fc3feeec2bff45c: Status 404 returned error can't find the container with id 26d89c0c0515f96c224e3b46815c7aacc546025c37ea61c18fc3feeec2bff45c
Jul 17 13:51:36.264122 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:36.264091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cr2qc" event={"ID":"8ec62e35-b2e6-4161-871f-3463c827961d","Type":"ContainerStarted","Data":"26d89c0c0515f96c224e3b46815c7aacc546025c37ea61c18fc3feeec2bff45c"}
Jul 17 13:51:37.269104 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:37.269072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cr2qc" event={"ID":"8ec62e35-b2e6-4161-871f-3463c827961d","Type":"ContainerStarted","Data":"74f83af14202894b4b56f16c3675a217899bee51b637bdc5529a2b091d89cf68"}
Jul 17 13:51:37.269502 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:37.269128    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cr2qc" event={"ID":"8ec62e35-b2e6-4161-871f-3463c827961d","Type":"ContainerStarted","Data":"cec191395209e6ebb8b93bbec4900ebdf16c7310130604f46e5c30c9480fe6a7"}
Jul 17 13:51:37.290445 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:37.287661    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cr2qc" podStartSLOduration=374.45221968 podStartE2EDuration="6m15.287640642s" totalImagesPullingTime="835.420962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:51:37.284737966 +0000 UTC m=+375.672085407" watchObservedRunningTime="2026-07-17 13:51:37.287640642 +0000 UTC m=+375.674988082"
Jul 17 13:51:47.313897 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.313863    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:51:47.314321 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.314138    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="prometheus" containerID="cri-o://bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0" gracePeriod=600
Jul 17 13:51:47.314321 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.314178    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy" containerID="cri-o://a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841" gracePeriod=600
Jul 17 13:51:47.314321 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.314233    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529" gracePeriod=600
Jul 17 13:51:47.314321 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.314185    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="thanos-sidecar" containerID="cri-o://5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b" gracePeriod=600
Jul 17 13:51:47.314537 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.314218    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-web" containerID="cri-o://d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1" gracePeriod=600
Jul 17 13:51:47.314537 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.314384    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="config-reloader" containerID="cri-o://5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8" gracePeriod=600
Jul 17 13:51:47.743774 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.743748    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:47.826550 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826520    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-kubelet-serving-ca-bundle\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826581    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-db\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826608    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-config-out\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826634    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-grpc-tls\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826649    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-2\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826675    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-metrics-client-certs\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826699    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-0\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826732    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-kube-rbac-proxy\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826757    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96km5\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-kube-api-access-96km5\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826787    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826824    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-thanos-prometheus-http-client-file\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826852    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-config\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826886    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-web-config\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.826931 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826910    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-serving-certs-ca-bundle\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.827320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826944    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-1\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.827320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826971    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-tls\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.827320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.826975    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:51:47.827320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.827006    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-tls-assets\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.827320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.827036    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-trusted-ca-bundle\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.827320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.827067    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-metrics-client-ca\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.827320 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.827104    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1d9e3383-f589-44d5-925e-db15a0c1b454\" (UID: \"1d9e3383-f589-44d5-925e-db15a0c1b454\") "
Jul 17 13:51:47.827667 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.827379    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.827667 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.827615    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:51:47.828169 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.827973    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:51:47.828531 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.828332    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:51:47.828622 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.828577    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:51:47.829354 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.829328    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:51:47.829455 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.829352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:51:47.829455 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.829358    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.829455 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.829425    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-config-out" (OuterVolumeSpecName: "config-out") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:51:47.829612 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.829459    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:51:47.829884 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.829848    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.830518 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.830489    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.830648 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.830616    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-kube-api-access-96km5" (OuterVolumeSpecName: "kube-api-access-96km5") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "kube-api-access-96km5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:51:47.830873 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.830856    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:51:47.830987 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.830963    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.831091 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.831065    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.831329 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.831306    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.831422 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.831397    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.831620 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.831607    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-config" (OuterVolumeSpecName: "config") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.841489 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.841439    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-web-config" (OuterVolumeSpecName: "web-config") pod "1d9e3383-f589-44d5-925e-db15a0c1b454" (UID: "1d9e3383-f589-44d5-925e-db15a0c1b454"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:51:47.928174 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928144    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-web-config\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928174 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928175    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928185    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928195    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928203    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-tls-assets\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928211    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928219    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-configmap-metrics-client-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928227    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928236    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-db\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928244    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9e3383-f589-44d5-925e-db15a0c1b454-config-out\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928253    2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-grpc-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928260    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928267    2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-metrics-client-certs\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928295    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9e3383-f589-44d5-925e-db15a0c1b454-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928304    2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-kube-rbac-proxy\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928312 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928312    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96km5\" (UniqueName: \"kubernetes.io/projected/1d9e3383-f589-44d5-925e-db15a0c1b454-kube-api-access-96km5\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928703 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928322    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928703 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928331    2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:47.928703 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:47.928339    2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9e3383-f589-44d5-925e-db15a0c1b454-config\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:51:48.317439 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317403    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529" exitCode=0
Jul 17 13:51:48.317439 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317432    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841" exitCode=0
Jul 17 13:51:48.317439 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317441    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1" exitCode=0
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317449    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b" exitCode=0
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317457    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8" exitCode=0
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317464    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0" exitCode=0
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317481    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"}
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317506    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317572    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"}
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317614    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"}
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317644    2575 scope.go:117] "RemoveContainer" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"}
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"}
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317738    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"}
Jul 17 13:51:48.317914 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.317776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9e3383-f589-44d5-925e-db15a0c1b454","Type":"ContainerDied","Data":"8be72f8f6077bc9663016402c8e42e5bf876a13ff8568f309695dafb62884138"}
Jul 17 13:51:48.325064 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.325045    2575 scope.go:117] "RemoveContainer" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.340993 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.340972    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:51:48.344247 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.344234    2575 scope.go:117] "RemoveContainer" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.348666 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.348641    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:51:48.351321 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.351260    2575 scope.go:117] "RemoveContainer" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.357501 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.357486    2575 scope.go:117] "RemoveContainer" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.363767 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.363752    2575 scope.go:117] "RemoveContainer" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.370363 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.370338    2575 scope.go:117] "RemoveContainer" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.376548 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.376530    2575 scope.go:117] "RemoveContainer" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.376849 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:51:48.376822    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": container with ID starting with 9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529 not found: ID does not exist" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.376910 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.376861    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"} err="failed to get container status \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": rpc error: code = NotFound desc = could not find container \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": container with ID starting with 9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529 not found: ID does not exist"
Jul 17 13:51:48.376910 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.376885    2575 scope.go:117] "RemoveContainer" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.377221 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:51:48.377201    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": container with ID starting with a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841 not found: ID does not exist" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.377292 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.377231    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"} err="failed to get container status \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": rpc error: code = NotFound desc = could not find container \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": container with ID starting with a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841 not found: ID does not exist"
Jul 17 13:51:48.377292 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.377251    2575 scope.go:117] "RemoveContainer" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.377544 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:51:48.377519    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": container with ID starting with d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1 not found: ID does not exist" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.377620 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.377551    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"} err="failed to get container status \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": rpc error: code = NotFound desc = could not find container \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": container with ID starting with d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1 not found: ID does not exist"
Jul 17 13:51:48.377620 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.377566    2575 scope.go:117] "RemoveContainer" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.377833 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:51:48.377815    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": container with ID starting with 5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b not found: ID does not exist" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.377891 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.377836    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"} err="failed to get container status \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": rpc error: code = NotFound desc = could not find container \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": container with ID starting with 5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b not found: ID does not exist"
Jul 17 13:51:48.377891 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.377848    2575 scope.go:117] "RemoveContainer" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.378086 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:51:48.378068    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": container with ID starting with 5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8 not found: ID does not exist" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.378124 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.378093    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"} err="failed to get container status \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": rpc error: code = NotFound desc = could not find container \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": container with ID starting with 5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8 not found: ID does not exist"
Jul 17 13:51:48.378124 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.378113    2575 scope.go:117] "RemoveContainer" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.378609 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:51:48.378585    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": container with ID starting with bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0 not found: ID does not exist" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.378699 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.378617    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"} err="failed to get container status \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": rpc error: code = NotFound desc = could not find container \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": container with ID starting with bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0 not found: ID does not exist"
Jul 17 13:51:48.378699 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.378637    2575 scope.go:117] "RemoveContainer" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.378871 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:51:48.378855    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": container with ID starting with 37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1 not found: ID does not exist" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.378933 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.378877    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"} err="failed to get container status \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": rpc error: code = NotFound desc = could not find container \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": container with ID starting with 37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1 not found: ID does not exist"
Jul 17 13:51:48.378933 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.378896    2575 scope.go:117] "RemoveContainer" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.379106 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379085    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"} err="failed to get container status \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": rpc error: code = NotFound desc = could not find container \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": container with ID starting with 9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529 not found: ID does not exist"
Jul 17 13:51:48.379157 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379107    2575 scope.go:117] "RemoveContainer" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.379346 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379329    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"} err="failed to get container status \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": rpc error: code = NotFound desc = could not find container \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": container with ID starting with a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841 not found: ID does not exist"
Jul 17 13:51:48.379410 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379348    2575 scope.go:117] "RemoveContainer" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.379531 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379515    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"} err="failed to get container status \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": rpc error: code = NotFound desc = could not find container \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": container with ID starting with d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1 not found: ID does not exist"
Jul 17 13:51:48.379599 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379532    2575 scope.go:117] "RemoveContainer" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.379789 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379770    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"} err="failed to get container status \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": rpc error: code = NotFound desc = could not find container \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": container with ID starting with 5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b not found: ID does not exist"
Jul 17 13:51:48.379862 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.379789    2575 scope.go:117] "RemoveContainer" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.380084 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380063    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"} err="failed to get container status \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": rpc error: code = NotFound desc = could not find container \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": container with ID starting with 5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8 not found: ID does not exist"
Jul 17 13:51:48.380164 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380085    2575 scope.go:117] "RemoveContainer" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.380469 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380446    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"} err="failed to get container status \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": rpc error: code = NotFound desc = could not find container \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": container with ID starting with bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0 not found: ID does not exist"
Jul 17 13:51:48.380469 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380469    2575 scope.go:117] "RemoveContainer" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.380713 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380694    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"} err="failed to get container status \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": rpc error: code = NotFound desc = could not find container \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": container with ID starting with 37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1 not found: ID does not exist"
Jul 17 13:51:48.380787 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380714    2575 scope.go:117] "RemoveContainer" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.380943 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380923    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"} err="failed to get container status \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": rpc error: code = NotFound desc = could not find container \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": container with ID starting with 9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529 not found: ID does not exist"
Jul 17 13:51:48.380996 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.380944    2575 scope.go:117] "RemoveContainer" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.381128 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381108    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"} err="failed to get container status \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": rpc error: code = NotFound desc = could not find container \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": container with ID starting with a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841 not found: ID does not exist"
Jul 17 13:51:48.381183 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381129    2575 scope.go:117] "RemoveContainer" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.381391 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381365    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"} err="failed to get container status \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": rpc error: code = NotFound desc = could not find container \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": container with ID starting with d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1 not found: ID does not exist"
Jul 17 13:51:48.381391 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381383    2575 scope.go:117] "RemoveContainer" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.381595 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381567    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"} err="failed to get container status \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": rpc error: code = NotFound desc = could not find container \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": container with ID starting with 5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b not found: ID does not exist"
Jul 17 13:51:48.381646 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381595    2575 scope.go:117] "RemoveContainer" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.381827 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381808    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"} err="failed to get container status \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": rpc error: code = NotFound desc = could not find container \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": container with ID starting with 5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8 not found: ID does not exist"
Jul 17 13:51:48.381865 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.381827    2575 scope.go:117] "RemoveContainer" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.382047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382027    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"} err="failed to get container status \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": rpc error: code = NotFound desc = could not find container \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": container with ID starting with bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0 not found: ID does not exist"
Jul 17 13:51:48.382095 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382047    2575 scope.go:117] "RemoveContainer" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.382230 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382215    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"} err="failed to get container status \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": rpc error: code = NotFound desc = could not find container \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": container with ID starting with 37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1 not found: ID does not exist"
Jul 17 13:51:48.382266 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382230    2575 scope.go:117] "RemoveContainer" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.382460 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382442    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"} err="failed to get container status \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": rpc error: code = NotFound desc = could not find container \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": container with ID starting with 9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529 not found: ID does not exist"
Jul 17 13:51:48.382460 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382460    2575 scope.go:117] "RemoveContainer" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.382667 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382649    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"} err="failed to get container status \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": rpc error: code = NotFound desc = could not find container \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": container with ID starting with a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841 not found: ID does not exist"
Jul 17 13:51:48.382703 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382668    2575 scope.go:117] "RemoveContainer" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.382855 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382838    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"} err="failed to get container status \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": rpc error: code = NotFound desc = could not find container \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": container with ID starting with d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1 not found: ID does not exist"
Jul 17 13:51:48.382912 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.382855    2575 scope.go:117] "RemoveContainer" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.383037 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383022    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"} err="failed to get container status \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": rpc error: code = NotFound desc = could not find container \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": container with ID starting with 5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b not found: ID does not exist"
Jul 17 13:51:48.383037 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383036    2575 scope.go:117] "RemoveContainer" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.383224 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383210    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"} err="failed to get container status \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": rpc error: code = NotFound desc = could not find container \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": container with ID starting with 5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8 not found: ID does not exist"
Jul 17 13:51:48.383261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383224    2575 scope.go:117] "RemoveContainer" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.383429 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383411    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"} err="failed to get container status \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": rpc error: code = NotFound desc = could not find container \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": container with ID starting with bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0 not found: ID does not exist"
Jul 17 13:51:48.383479 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383429    2575 scope.go:117] "RemoveContainer" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.383615 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383599    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"} err="failed to get container status \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": rpc error: code = NotFound desc = could not find container \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": container with ID starting with 37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1 not found: ID does not exist"
Jul 17 13:51:48.383654 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383615    2575 scope.go:117] "RemoveContainer" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.383795 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383781    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"} err="failed to get container status \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": rpc error: code = NotFound desc = could not find container \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": container with ID starting with 9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529 not found: ID does not exist"
Jul 17 13:51:48.383839 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383795    2575 scope.go:117] "RemoveContainer" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.383978 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383962    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"} err="failed to get container status \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": rpc error: code = NotFound desc = could not find container \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": container with ID starting with a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841 not found: ID does not exist"
Jul 17 13:51:48.383978 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.383977    2575 scope.go:117] "RemoveContainer" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.384157 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384141    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"} err="failed to get container status \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": rpc error: code = NotFound desc = could not find container \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": container with ID starting with d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1 not found: ID does not exist"
Jul 17 13:51:48.384196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384159    2575 scope.go:117] "RemoveContainer" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.384362 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384344    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"} err="failed to get container status \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": rpc error: code = NotFound desc = could not find container \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": container with ID starting with 5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b not found: ID does not exist"
Jul 17 13:51:48.384362 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384359    2575 scope.go:117] "RemoveContainer" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.384537 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384521    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"} err="failed to get container status \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": rpc error: code = NotFound desc = could not find container \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": container with ID starting with 5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8 not found: ID does not exist"
Jul 17 13:51:48.384578 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384537    2575 scope.go:117] "RemoveContainer" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.384718 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384702    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"} err="failed to get container status \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": rpc error: code = NotFound desc = could not find container \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": container with ID starting with bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0 not found: ID does not exist"
Jul 17 13:51:48.384755 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384718    2575 scope.go:117] "RemoveContainer" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.384889 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384873    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"} err="failed to get container status \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": rpc error: code = NotFound desc = could not find container \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": container with ID starting with 37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1 not found: ID does not exist"
Jul 17 13:51:48.384889 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.384889    2575 scope.go:117] "RemoveContainer" containerID="9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"
Jul 17 13:51:48.385052 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385039    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529"} err="failed to get container status \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": rpc error: code = NotFound desc = could not find container \"9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529\": container with ID starting with 9c3d159471432740cf81c226af7411d55c4b63384524a9a34be77952f8934529 not found: ID does not exist"
Jul 17 13:51:48.385095 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385053    2575 scope.go:117] "RemoveContainer" containerID="a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"
Jul 17 13:51:48.385263 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385245    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841"} err="failed to get container status \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": rpc error: code = NotFound desc = could not find container \"a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841\": container with ID starting with a40537b3a1cae081a2772fcbc79b1fd7df2db84579f5bd19879a8bf412011841 not found: ID does not exist"
Jul 17 13:51:48.385380 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385263    2575 scope.go:117] "RemoveContainer" containerID="d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"
Jul 17 13:51:48.385499 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385482    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1"} err="failed to get container status \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": rpc error: code = NotFound desc = could not find container \"d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1\": container with ID starting with d62cfaed3acced75ef0ff569a06527a3fb84c3dbb001c720077e7f8dc3d1e1c1 not found: ID does not exist"
Jul 17 13:51:48.385545 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385499    2575 scope.go:117] "RemoveContainer" containerID="5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"
Jul 17 13:51:48.385692 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385674    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b"} err="failed to get container status \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": rpc error: code = NotFound desc = could not find container \"5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b\": container with ID starting with 5bbeabe6bf715ac71250c279efb5d5cb637a58001a869c219554d8e242deac2b not found: ID does not exist"
Jul 17 13:51:48.385728 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385692    2575 scope.go:117] "RemoveContainer" containerID="5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"
Jul 17 13:51:48.385895 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385879    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8"} err="failed to get container status \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": rpc error: code = NotFound desc = could not find container \"5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8\": container with ID starting with 5bc9359b70aabae1dbf1997438189fbcb44f21f8d9ef02a624dc8565f2f5f9f8 not found: ID does not exist"
Jul 17 13:51:48.385934 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.385895    2575 scope.go:117] "RemoveContainer" containerID="bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"
Jul 17 13:51:48.386096 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.386078    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0"} err="failed to get container status \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": rpc error: code = NotFound desc = could not find container \"bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0\": container with ID starting with bf4ea66c1a234e71fb904d7c63614b5f8c25d26edc047a3696a0228a0fdffae0 not found: ID does not exist"
Jul 17 13:51:48.386096 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.386095    2575 scope.go:117] "RemoveContainer" containerID="37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"
Jul 17 13:51:48.386305 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.386271    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1"} err="failed to get container status \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": rpc error: code = NotFound desc = could not find container \"37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1\": container with ID starting with 37852c34fdfc96fcff5ed47355d126e1cd9b1bb8ea60ddd2df033104b8dce2d1 not found: ID does not exist"
Jul 17 13:51:48.389716 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.389698    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:51:48.390008 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.389975    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-web"
Jul 17 13:51:48.390008 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.389988    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-web"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390022    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390033    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390046    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="init-config-reloader"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390053    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="init-config-reloader"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390061    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="config-reloader"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390069    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="config-reloader"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390083    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="prometheus"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390091    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="prometheus"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390099    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="thanos-sidecar"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390106    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="thanos-sidecar"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390115    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-thanos"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390122    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-thanos"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390139    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d6ef3b-0c52-4586-a224-7943c1b53190" containerName="registry"
Jul 17 13:51:48.390182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390147    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d6ef3b-0c52-4586-a224-7943c1b53190" containerName="registry"
Jul 17 13:51:48.390587 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390218    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy"
Jul 17 13:51:48.390587 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390226    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d6ef3b-0c52-4586-a224-7943c1b53190" containerName="registry"
Jul 17 13:51:48.390587 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390233    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="prometheus"
Jul 17 13:51:48.390587 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390239    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="config-reloader"
Jul 17 13:51:48.390587 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390244    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-web"
Jul 17 13:51:48.390587 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390250    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="thanos-sidecar"
Jul 17 13:51:48.390587 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.390256    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" containerName="kube-rbac-proxy-thanos"
Jul 17 13:51:48.395148 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.395132    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.400226 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.400200    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 13:51:48.400326 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.400310    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r5p5l\""
Jul 17 13:51:48.400397 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.400337    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 13:51:48.400497 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.400482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-78j28bmbmr6n0\""
Jul 17 13:51:48.401697 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.401679    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 13:51:48.401785 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.401685    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 13:51:48.401945 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.401930    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 13:51:48.401945 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.401937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 13:51:48.402134 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.402016    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 13:51:48.402134 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.402022    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 13:51:48.402134 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.402074    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 13:51:48.403974 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.403958    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 13:51:48.404163 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.404149    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 13:51:48.404742 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.404724    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 13:51:48.404742 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.404736    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 13:51:48.407331 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.407312    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 13:51:48.408149 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.408134    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 13:51:48.418261 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.418236    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:51:48.532599 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532574    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532694 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532694 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-config\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532694 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532639    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532694 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532812 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-kube-api-access-wnr2s\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532812 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532812 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532812 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532787    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532812 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532948 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532948 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.532948 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.533032 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.533032 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.533032 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.532997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.533032 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.533013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.533032 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.533028    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.533161 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.533042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.533161 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.533074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634314 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634232    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634314 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634314 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634492 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634492 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634492 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634492 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634492 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-config\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-kube-api-access-wnr2s\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.634717 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.635094 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634742    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.635094 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.635094 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.635094 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.635094 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.635094 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.634987    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.635469 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.635422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.637293 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.636837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.637293 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.637014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.637293 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.637227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.637830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.637318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.637830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.637681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.637830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.637765    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.637830 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.637820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.638848 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.638822    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.639041 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.639021    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.639759 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.639734    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.640292 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.640257    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.640431 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.640324    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.640431 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.640363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.640431 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.640386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.641029 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.641012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.641216 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.641197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-config\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.644572 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.644548    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8-kube-api-access-wnr2s\") pod \"prometheus-k8s-0\" (UID: \"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.703873 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.703855    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:51:48.835611 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:48.835581    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 13:51:48.839385 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:51:48.839355    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5dcc22_fbb2_4273_91c7_3f5e73bd03b8.slice/crio-e397ad4e6b903232033535c21419204fac9ff5120dc4058e123e9ba6d666d074 WatchSource:0}: Error finding container e397ad4e6b903232033535c21419204fac9ff5120dc4058e123e9ba6d666d074: Status 404 returned error can't find the container with id e397ad4e6b903232033535c21419204fac9ff5120dc4058e123e9ba6d666d074
Jul 17 13:51:49.324457 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:49.324427    2575 generic.go:356] "Generic (PLEG): container finished" podID="9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8" containerID="cc5552697d4e5dd7e3f9a19d710a21e42d0e0c3cbfe3b0e14a53c114610c2d2e" exitCode=0
Jul 17 13:51:49.324825 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:49.324471    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerDied","Data":"cc5552697d4e5dd7e3f9a19d710a21e42d0e0c3cbfe3b0e14a53c114610c2d2e"}
Jul 17 13:51:49.324825 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:49.324524    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerStarted","Data":"e397ad4e6b903232033535c21419204fac9ff5120dc4058e123e9ba6d666d074"}
Jul 17 13:51:50.073898 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.073868    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9e3383-f589-44d5-925e-db15a0c1b454" path="/var/lib/kubelet/pods/1d9e3383-f589-44d5-925e-db15a0c1b454/volumes"
Jul 17 13:51:50.332715 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.332640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerStarted","Data":"f153294c6ee1642267d898eee7ae84fe954eebf729515a195c53a75a101b152c"}
Jul 17 13:51:50.333047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.332723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerStarted","Data":"df1d9d4d4bcfe538864f76a406b724077b74a9e60435f62d5ff84a7f0f274334"}
Jul 17 13:51:50.333047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.332761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerStarted","Data":"9cb20628ebfc10037169db23cc340b1d70703afabf96827a15e4ee807b216371"}
Jul 17 13:51:50.333047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.332798    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerStarted","Data":"a5b1d7d7ed71d258792250975d1866570b4bd5eb14a2919bedf1e2cdcc3e6015"}
Jul 17 13:51:50.333047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.332835    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerStarted","Data":"1eff0b03fbd878c3819b662e29ca0d707d4a9b79346993d24e516c8c31d3c84c"}
Jul 17 13:51:50.333047 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.332868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8","Type":"ContainerStarted","Data":"4a80e2ead4db7e3f88d37607ec7cfed3d68e08b26576437556406500ce4cdd63"}
Jul 17 13:51:50.363001 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:50.362952    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.344232924 podStartE2EDuration="2.362934501s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.701577ms" isStatefulPod=false podCreationTimestamp="2026-07-17 13:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:51:50.360826926 +0000 UTC m=+388.748174357" watchObservedRunningTime="2026-07-17 13:51:50.362934501 +0000 UTC m=+388.750281940"
Jul 17 13:51:53.704257 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:51:53.704219    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:52:48.704919 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:52:48.704814    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:52:48.721093 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:52:48.721064    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:52:49.581849 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:52:49.581814    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 13:53:15.888087 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.888047    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"]
Jul 17 13:53:15.891162 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.891145    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:15.893865 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.893834    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2s75k\""
Jul 17 13:53:15.893865 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.893844    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 13:53:15.894603 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.894588    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 13:53:15.901359 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.901332    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"]
Jul 17 13:53:15.932442 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.932422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgj8q\" (UniqueName: \"kubernetes.io/projected/a30e3b17-258b-4a58-866a-edeb85970e0b-kube-api-access-sgj8q\") pod \"cert-manager-cainjector-559cdd8c87-tkbc8\" (UID: \"a30e3b17-258b-4a58-866a-edeb85970e0b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:15.932537 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:15.932459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a30e3b17-258b-4a58-866a-edeb85970e0b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tkbc8\" (UID: \"a30e3b17-258b-4a58-866a-edeb85970e0b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:16.033314 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:16.033294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a30e3b17-258b-4a58-866a-edeb85970e0b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tkbc8\" (UID: \"a30e3b17-258b-4a58-866a-edeb85970e0b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:16.033398 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:16.033350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgj8q\" (UniqueName: \"kubernetes.io/projected/a30e3b17-258b-4a58-866a-edeb85970e0b-kube-api-access-sgj8q\") pod \"cert-manager-cainjector-559cdd8c87-tkbc8\" (UID: \"a30e3b17-258b-4a58-866a-edeb85970e0b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:16.049494 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:16.049476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgj8q\" (UniqueName: \"kubernetes.io/projected/a30e3b17-258b-4a58-866a-edeb85970e0b-kube-api-access-sgj8q\") pod \"cert-manager-cainjector-559cdd8c87-tkbc8\" (UID: \"a30e3b17-258b-4a58-866a-edeb85970e0b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:16.058469 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:16.058444    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a30e3b17-258b-4a58-866a-edeb85970e0b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tkbc8\" (UID: \"a30e3b17-258b-4a58-866a-edeb85970e0b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:16.217885 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:16.217863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"
Jul 17 13:53:16.338058 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:16.338031    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8"]
Jul 17 13:53:16.341002 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:53:16.340979    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda30e3b17_258b_4a58_866a_edeb85970e0b.slice/crio-80c6e5f78273b821e7af45d59fd927faae1cdfa3f73c7716c57f8d6559dc1828 WatchSource:0}: Error finding container 80c6e5f78273b821e7af45d59fd927faae1cdfa3f73c7716c57f8d6559dc1828: Status 404 returned error can't find the container with id 80c6e5f78273b821e7af45d59fd927faae1cdfa3f73c7716c57f8d6559dc1828
Jul 17 13:53:16.678390 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:16.678333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8" event={"ID":"a30e3b17-258b-4a58-866a-edeb85970e0b","Type":"ContainerStarted","Data":"80c6e5f78273b821e7af45d59fd927faae1cdfa3f73c7716c57f8d6559dc1828"}
Jul 17 13:53:19.693453 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:19.693364    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8" event={"ID":"a30e3b17-258b-4a58-866a-edeb85970e0b","Type":"ContainerStarted","Data":"67e533d6665675f038d057a3d5774df7ff41fe9f86dd3e9ffd9e931c381d866c"}
Jul 17 13:53:19.715077 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:19.715034    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tkbc8" podStartSLOduration=1.642934301 podStartE2EDuration="4.715000634s" totalImagesPullingTime="3.072066333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:53:19.71399745 +0000 UTC m=+478.101344890" watchObservedRunningTime="2026-07-17 13:53:19.715000634 +0000 UTC m=+478.102348075"
Jul 17 13:53:26.279822 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.279769    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-44ms9"]
Jul 17 13:53:26.283450 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.283428    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.285614 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.285580    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-52skd\""
Jul 17 13:53:26.292195 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.292167    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-44ms9"]
Jul 17 13:53:26.418664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.418618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff310c38-c17a-40d3-b746-90e0edda18f8-bound-sa-token\") pod \"cert-manager-55b4d954d5-44ms9\" (UID: \"ff310c38-c17a-40d3-b746-90e0edda18f8\") " pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.418664 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.418668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzvj\" (UniqueName: \"kubernetes.io/projected/ff310c38-c17a-40d3-b746-90e0edda18f8-kube-api-access-8dzvj\") pod \"cert-manager-55b4d954d5-44ms9\" (UID: \"ff310c38-c17a-40d3-b746-90e0edda18f8\") " pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.519197 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.519169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff310c38-c17a-40d3-b746-90e0edda18f8-bound-sa-token\") pod \"cert-manager-55b4d954d5-44ms9\" (UID: \"ff310c38-c17a-40d3-b746-90e0edda18f8\") " pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.519419 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.519214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzvj\" (UniqueName: \"kubernetes.io/projected/ff310c38-c17a-40d3-b746-90e0edda18f8-kube-api-access-8dzvj\") pod \"cert-manager-55b4d954d5-44ms9\" (UID: \"ff310c38-c17a-40d3-b746-90e0edda18f8\") " pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.528019 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.527980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff310c38-c17a-40d3-b746-90e0edda18f8-bound-sa-token\") pod \"cert-manager-55b4d954d5-44ms9\" (UID: \"ff310c38-c17a-40d3-b746-90e0edda18f8\") " pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.528219 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.528180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzvj\" (UniqueName: \"kubernetes.io/projected/ff310c38-c17a-40d3-b746-90e0edda18f8-kube-api-access-8dzvj\") pod \"cert-manager-55b4d954d5-44ms9\" (UID: \"ff310c38-c17a-40d3-b746-90e0edda18f8\") " pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.595072 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.595012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-44ms9"
Jul 17 13:53:26.721661 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:26.721636    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-44ms9"]
Jul 17 13:53:26.723759 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:53:26.723734    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff310c38_c17a_40d3_b746_90e0edda18f8.slice/crio-f4b909a1cb76206f61c1b0163be58318607987ecc2ec9cab8cacb74a1bcb5fae WatchSource:0}: Error finding container f4b909a1cb76206f61c1b0163be58318607987ecc2ec9cab8cacb74a1bcb5fae: Status 404 returned error can't find the container with id f4b909a1cb76206f61c1b0163be58318607987ecc2ec9cab8cacb74a1bcb5fae
Jul 17 13:53:27.727539 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:27.727501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-44ms9" event={"ID":"ff310c38-c17a-40d3-b746-90e0edda18f8","Type":"ContainerStarted","Data":"ab827046c5334a05ca236b861f9b1e7d6fdda5d83f1e3f3eaed0862532a45dfc"}
Jul 17 13:53:27.727925 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:27.727549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-44ms9" event={"ID":"ff310c38-c17a-40d3-b746-90e0edda18f8","Type":"ContainerStarted","Data":"f4b909a1cb76206f61c1b0163be58318607987ecc2ec9cab8cacb74a1bcb5fae"}
Jul 17 13:53:27.744365 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:27.744318    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-44ms9" podStartSLOduration=1.744303183 podStartE2EDuration="1.744303183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:53:27.743035751 +0000 UTC m=+486.130383190" watchObservedRunningTime="2026-07-17 13:53:27.744303183 +0000 UTC m=+486.131650624"
Jul 17 13:53:49.044492 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.044452    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"]
Jul 17 13:53:49.047845 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.047820    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.051681 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.051657    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 13:53:49.051805 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.051726    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 13:53:49.051805 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.051737    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 13:53:49.051805 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.051780    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pxqwq\""
Jul 17 13:53:49.052160 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.052144    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 13:53:49.094928 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.094904    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"]
Jul 17 13:53:49.189576 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.189549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8v24\" (UniqueName: \"kubernetes.io/projected/837e5de8-0d6c-47c9-a22d-a684cefaca39-kube-api-access-f8v24\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.189693 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.189588    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/837e5de8-0d6c-47c9-a22d-a684cefaca39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.189693 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.189608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837e5de8-0d6c-47c9-a22d-a684cefaca39-tmp\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.189693 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.189628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/837e5de8-0d6c-47c9-a22d-a684cefaca39-manifests\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.189836 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.189710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/837e5de8-0d6c-47c9-a22d-a684cefaca39-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.290333 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.290312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/837e5de8-0d6c-47c9-a22d-a684cefaca39-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.290455 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.290352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8v24\" (UniqueName: \"kubernetes.io/projected/837e5de8-0d6c-47c9-a22d-a684cefaca39-kube-api-access-f8v24\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.290455 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.290381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/837e5de8-0d6c-47c9-a22d-a684cefaca39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.290455 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.290401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837e5de8-0d6c-47c9-a22d-a684cefaca39-tmp\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.290455 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.290420    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/837e5de8-0d6c-47c9-a22d-a684cefaca39-manifests\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.290785 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.290766    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/837e5de8-0d6c-47c9-a22d-a684cefaca39-manifests\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.292862 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.292808    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837e5de8-0d6c-47c9-a22d-a684cefaca39-tmp\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.292974 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.292954    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/837e5de8-0d6c-47c9-a22d-a684cefaca39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.293060 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.293042    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/837e5de8-0d6c-47c9-a22d-a684cefaca39-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.300384 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.300333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8v24\" (UniqueName: \"kubernetes.io/projected/837e5de8-0d6c-47c9-a22d-a684cefaca39-kube-api-access-f8v24\") pod \"opendatahub-operator-controller-manager-57f79678b4-h4wpw\" (UID: \"837e5de8-0d6c-47c9-a22d-a684cefaca39\") " pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.359046 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.359021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:49.487202 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.487178    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"]
Jul 17 13:53:49.488915 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:53:49.488891    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837e5de8_0d6c_47c9_a22d_a684cefaca39.slice/crio-9aa6bac8ba3a89f82fb8958319c035970fcc2e1adc6784dad2847e9ada04c9a5 WatchSource:0}: Error finding container 9aa6bac8ba3a89f82fb8958319c035970fcc2e1adc6784dad2847e9ada04c9a5: Status 404 returned error can't find the container with id 9aa6bac8ba3a89f82fb8958319c035970fcc2e1adc6784dad2847e9ada04c9a5
Jul 17 13:53:49.820818 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:49.820783    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw" event={"ID":"837e5de8-0d6c-47c9-a22d-a684cefaca39","Type":"ContainerStarted","Data":"9aa6bac8ba3a89f82fb8958319c035970fcc2e1adc6784dad2847e9ada04c9a5"}
Jul 17 13:53:52.836636 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:52.836602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw" event={"ID":"837e5de8-0d6c-47c9-a22d-a684cefaca39","Type":"ContainerStarted","Data":"b9a4e1e4bed4f0e2dfdf16d69beb43bb28fcccf065352e0d2220fa9c0561099e"}
Jul 17 13:53:53.842980 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:53.842949    2575 generic.go:356] "Generic (PLEG): container finished" podID="837e5de8-0d6c-47c9-a22d-a684cefaca39" containerID="b9a4e1e4bed4f0e2dfdf16d69beb43bb28fcccf065352e0d2220fa9c0561099e" exitCode=0
Jul 17 13:53:53.843376 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:53.843018    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw" event={"ID":"837e5de8-0d6c-47c9-a22d-a684cefaca39","Type":"ContainerDied","Data":"b9a4e1e4bed4f0e2dfdf16d69beb43bb28fcccf065352e0d2220fa9c0561099e"}
Jul 17 13:53:54.848932 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:54.848902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw" event={"ID":"837e5de8-0d6c-47c9-a22d-a684cefaca39","Type":"ContainerStarted","Data":"270823e0a6842ea92fee68247f91e078e2c4b29ee2999e3e61ee96518daf0be3"}
Jul 17 13:53:54.849341 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:54.849010    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:53:54.870006 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:53:54.869967    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw" podStartSLOduration=2.595048244 podStartE2EDuration="6.86995121s" totalImagesPullingTime="3.222643052s" totalInitContainerRuntime="1.052259914s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:53:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:53:54.869791813 +0000 UTC m=+513.257139252" watchObservedRunningTime="2026-07-17 13:53:54.86995121 +0000 UTC m=+513.257298649"
Jul 17 13:54:05.855384 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:05.855351    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57f79678b4-h4wpw"
Jul 17 13:54:06.039877 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.039837    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"]
Jul 17 13:54:06.048330 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.048304    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.050906 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.050880    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jpqgw\""
Jul 17 13:54:06.051243 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.051200    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 13:54:06.051243 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.051235    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 13:54:06.051420 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.051258    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 13:54:06.051420 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.051316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"]
Jul 17 13:54:06.051528 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.051518    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 13:54:06.051582 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.051560    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 13:54:06.118448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.118402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6sc\" (UniqueName: \"kubernetes.io/projected/cf0cedd5-727b-48a9-af08-e59b4b26f755-kube-api-access-kw6sc\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.118448 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.118450    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf0cedd5-727b-48a9-af08-e59b4b26f755-metrics-cert\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.118581 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.118531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf0cedd5-727b-48a9-af08-e59b4b26f755-manager-config\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.118649 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.118631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf0cedd5-727b-48a9-af08-e59b4b26f755-cert\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.219246 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.219225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf0cedd5-727b-48a9-af08-e59b4b26f755-metrics-cert\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.219393 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.219297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf0cedd5-727b-48a9-af08-e59b4b26f755-manager-config\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.219393 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.219349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf0cedd5-727b-48a9-af08-e59b4b26f755-cert\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.219506 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.219400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6sc\" (UniqueName: \"kubernetes.io/projected/cf0cedd5-727b-48a9-af08-e59b4b26f755-kube-api-access-kw6sc\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.219948 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.219926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf0cedd5-727b-48a9-af08-e59b4b26f755-manager-config\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.221643 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.221624    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf0cedd5-727b-48a9-af08-e59b4b26f755-cert\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.221726 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.221656    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf0cedd5-727b-48a9-af08-e59b4b26f755-metrics-cert\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.227734 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.227716    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6sc\" (UniqueName: \"kubernetes.io/projected/cf0cedd5-727b-48a9-af08-e59b4b26f755-kube-api-access-kw6sc\") pod \"lws-controller-manager-77668bc899-lbr4h\" (UID: \"cf0cedd5-727b-48a9-af08-e59b4b26f755\") " pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.357686 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.357662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:06.482557 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.482527    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"]
Jul 17 13:54:06.485414 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:54:06.485390    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0cedd5_727b_48a9_af08_e59b4b26f755.slice/crio-df4e112a6a4da4be00491944aa1b8cc423eb1b706fbbd8c0f696b19426edc3bf WatchSource:0}: Error finding container df4e112a6a4da4be00491944aa1b8cc423eb1b706fbbd8c0f696b19426edc3bf: Status 404 returned error can't find the container with id df4e112a6a4da4be00491944aa1b8cc423eb1b706fbbd8c0f696b19426edc3bf
Jul 17 13:54:06.903163 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:06.903131    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h" event={"ID":"cf0cedd5-727b-48a9-af08-e59b4b26f755","Type":"ContainerStarted","Data":"df4e112a6a4da4be00491944aa1b8cc423eb1b706fbbd8c0f696b19426edc3bf"}
Jul 17 13:54:08.914939 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:08.914908    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h" event={"ID":"cf0cedd5-727b-48a9-af08-e59b4b26f755","Type":"ContainerStarted","Data":"05fc817142ff1b51a1b7905a98acfc6a608e49d52a835303ff65e48f5fe7181c"}
Jul 17 13:54:08.915333 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:08.914996    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:08.941645 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:08.941195    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h" podStartSLOduration=0.674479162 podStartE2EDuration="2.941178104s" totalImagesPullingTime="2.266698942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:54:08.939624383 +0000 UTC m=+527.326971826" watchObservedRunningTime="2026-07-17 13:54:08.941178104 +0000 UTC m=+527.328525543"
Jul 17 13:54:18.028678 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.028585    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"]
Jul 17 13:54:18.032103 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.032085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.035412 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.035389    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 13:54:18.035412 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.035389    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 13:54:18.038776 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.038442    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 13:54:18.038889 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.038869    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 13:54:18.039155 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.039136    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rl9k4\""
Jul 17 13:54:18.039239 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.039139    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 13:54:18.039974 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.039955    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 13:54:18.067151 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.067129    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"]
Jul 17 13:54:18.101481 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.101462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbxt\" (UniqueName: \"kubernetes.io/projected/525694bd-6c3e-4648-81db-40546200adc4-kube-api-access-fvbxt\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.101567 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.101493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/525694bd-6c3e-4648-81db-40546200adc4-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.101567 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.101519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/525694bd-6c3e-4648-81db-40546200adc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.101635 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.101582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525694bd-6c3e-4648-81db-40546200adc4-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.101635 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.101624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.101728 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.101711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.101762 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.101735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.202718 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.202693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525694bd-6c3e-4648-81db-40546200adc4-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.202820 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.202748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.202820 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.202800    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.202921 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.202825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.202921 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.202850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbxt\" (UniqueName: \"kubernetes.io/projected/525694bd-6c3e-4648-81db-40546200adc4-kube-api-access-fvbxt\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.202921 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.202886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/525694bd-6c3e-4648-81db-40546200adc4-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.203061 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.202924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/525694bd-6c3e-4648-81db-40546200adc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.203944 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.203921    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/525694bd-6c3e-4648-81db-40546200adc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.205481 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.205461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.205748 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.205711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.205976 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.205942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/525694bd-6c3e-4648-81db-40546200adc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.206099 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.206080    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/525694bd-6c3e-4648-81db-40546200adc4-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.216627 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.216604    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525694bd-6c3e-4648-81db-40546200adc4-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.217271 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.217250    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbxt\" (UniqueName: \"kubernetes.io/projected/525694bd-6c3e-4648-81db-40546200adc4-kube-api-access-fvbxt\") pod \"istiod-openshift-gateway-94bb8fbfd-x4zlz\" (UID: \"525694bd-6c3e-4648-81db-40546200adc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.342035 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.341988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:18.480010 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.479984    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"]
Jul 17 13:54:18.482214 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:54:18.482184    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525694bd_6c3e_4648_81db_40546200adc4.slice/crio-b1478c0e8abcfd0d1127a7758dfbfcd1b7454c058ca7820e58fbffccca33e128 WatchSource:0}: Error finding container b1478c0e8abcfd0d1127a7758dfbfcd1b7454c058ca7820e58fbffccca33e128: Status 404 returned error can't find the container with id b1478c0e8abcfd0d1127a7758dfbfcd1b7454c058ca7820e58fbffccca33e128
Jul 17 13:54:18.966890 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:18.966860    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz" event={"ID":"525694bd-6c3e-4648-81db-40546200adc4","Type":"ContainerStarted","Data":"b1478c0e8abcfd0d1127a7758dfbfcd1b7454c058ca7820e58fbffccca33e128"}
Jul 17 13:54:19.922107 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:19.922076    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77668bc899-lbr4h"
Jul 17 13:54:20.701054 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:20.701016    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 13:54:20.701187 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:20.701081    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 13:54:20.979196 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:20.979170    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz" event={"ID":"525694bd-6c3e-4648-81db-40546200adc4","Type":"ContainerStarted","Data":"6fc75c1a74808e3c6b5bbe9eb9f520d18416d400b1140297004fa4edc55147b7"}
Jul 17 13:54:20.979693 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:20.979303    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:54:21.045227 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:21.045182    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz" podStartSLOduration=1.828499736 podStartE2EDuration="4.045164164s" totalImagesPullingTime="2.216664428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:54:21.042712139 +0000 UTC m=+539.430059579" watchObservedRunningTime="2026-07-17 13:54:21.045164164 +0000 UTC m=+539.432511604"
Jul 17 13:54:21.984410 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:54:21.984380    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-x4zlz"
Jul 17 13:55:21.975542 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:55:21.975508    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 13:55:21.977578 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:55:21.977554    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 13:55:21.979510 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:55:21.979486    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 13:55:21.981822 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:55:21.981790    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 13:55:21.987025 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:55:21.987000    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:55:21.988983 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:55:21.988966    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 13:56:05.487786 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.487747    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"]
Jul 17 13:56:05.490991 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.490972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.493572 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.493551    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bzpxn\""
Jul 17 13:56:05.493693 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.493587    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 13:56:05.493871 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.493856    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 13:56:05.512745 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.512721    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"]
Jul 17 13:56:05.585404 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.585382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.585508 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.585423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lrj\" (UniqueName: \"kubernetes.io/projected/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-kube-api-access-k4lrj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.686610 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.686586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.686706 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.686622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lrj\" (UniqueName: \"kubernetes.io/projected/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-kube-api-access-k4lrj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.686957 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.686936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.695235 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.695214    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lrj\" (UniqueName: \"kubernetes.io/projected/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-kube-api-access-k4lrj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.801647 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.801599    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:05.928381 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.928354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"]
Jul 17 13:56:05.931786 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:56:05.931763    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd5b617_ddc5_4d79_8d5a_7d5d39c7de24.slice/crio-aafa9738555effe62b37c87da3442a537321ae4c824d7c2b25650cc892d41375 WatchSource:0}: Error finding container aafa9738555effe62b37c87da3442a537321ae4c824d7c2b25650cc892d41375: Status 404 returned error can't find the container with id aafa9738555effe62b37c87da3442a537321ae4c824d7c2b25650cc892d41375
Jul 17 13:56:05.934263 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:05.934249    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:56:06.287239 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.287212    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"]
Jul 17 13:56:06.296334 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.294577    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"]
Jul 17 13:56:06.329027 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.328998    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"]
Jul 17 13:56:06.333355 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.333333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.358237 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.358213    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"]
Jul 17 13:56:06.394656 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.394625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6w5s\" (UniqueName: \"kubernetes.io/projected/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-kube-api-access-c6w5s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gmgx6\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.394751 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.394691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gmgx6\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.495860 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.495835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6w5s\" (UniqueName: \"kubernetes.io/projected/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-kube-api-access-c6w5s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gmgx6\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.496242 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.495921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gmgx6\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.496338 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.496319    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gmgx6\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.504459 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.504437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6w5s\" (UniqueName: \"kubernetes.io/projected/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-kube-api-access-c6w5s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gmgx6\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.645363 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.645230    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:06.897561 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:06.897535    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"]
Jul 17 13:56:06.975980 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:56:06.975936    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a043e5_99ce_48d3_8ddb_7b4df2308ec5.slice/crio-668f1508ec42ff61d12a78e3eb8e150988a3cafd2a8b4162d9a5d4c0e14ef2ef WatchSource:0}: Error finding container 668f1508ec42ff61d12a78e3eb8e150988a3cafd2a8b4162d9a5d4c0e14ef2ef: Status 404 returned error can't find the container with id 668f1508ec42ff61d12a78e3eb8e150988a3cafd2a8b4162d9a5d4c0e14ef2ef
Jul 17 13:56:07.473360 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:07.473318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6" event={"ID":"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5","Type":"ContainerStarted","Data":"668f1508ec42ff61d12a78e3eb8e150988a3cafd2a8b4162d9a5d4c0e14ef2ef"}
Jul 17 13:56:10.491836 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.491804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6" event={"ID":"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5","Type":"ContainerStarted","Data":"b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f"}
Jul 17 13:56:10.492379 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.491868    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:10.493447 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.493422    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" containerName="manager" containerID="cri-o://437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87" gracePeriod=2
Jul 17 13:56:10.514193 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.514155    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6" podStartSLOduration=1.227617537 podStartE2EDuration="4.51414069s" totalImagesPullingTime="3.286523153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:56:10.512379342 +0000 UTC m=+648.899726781" watchObservedRunningTime="2026-07-17 13:56:10.51414069 +0000 UTC m=+648.901488128"
Jul 17 13:56:10.727032 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.727004    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:10.729266 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:56:10.729238    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" is forbidden: User \"system:node:ip-10-0-129-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-191.ec2.internal' and this object" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:10.833825 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.833762    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-extensions-socket-volume\") pod \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") "
Jul 17 13:56:10.833825 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.833821    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4lrj\" (UniqueName: \"kubernetes.io/projected/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-kube-api-access-k4lrj\") pod \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\" (UID: \"7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24\") "
Jul 17 13:56:10.834073 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.834041    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" (UID: "7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:56:10.834177 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.834164    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-extensions-socket-volume\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:56:10.835823 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.835805    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-kube-api-access-k4lrj" (OuterVolumeSpecName: "kube-api-access-k4lrj") pod "7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" (UID: "7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24"). InnerVolumeSpecName "kube-api-access-k4lrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:56:10.934752 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:10.934727    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4lrj\" (UniqueName: \"kubernetes.io/projected/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24-kube-api-access-k4lrj\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:56:11.499944 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:11.499902    2575 generic.go:356] "Generic (PLEG): container finished" podID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" containerID="437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87" exitCode=2
Jul 17 13:56:11.500439 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:11.499968    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:11.500439 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:11.500018    2575 scope.go:117] "RemoveContainer" containerID="437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87"
Jul 17 13:56:11.502222 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:56:11.502191    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" is forbidden: User \"system:node:ip-10-0-129-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-191.ec2.internal' and this object" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:11.509633 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:11.509609    2575 scope.go:117] "RemoveContainer" containerID="437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87"
Jul 17 13:56:11.509948 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:56:11.509928    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87\": container with ID starting with 437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87 not found: ID does not exist" containerID="437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87"
Jul 17 13:56:11.510025 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:11.509960    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87"} err="failed to get container status \"437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87\": rpc error: code = NotFound desc = could not find container \"437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87\": container with ID starting with 437a3697d4d6524097e4c3e2a3d98178cbfb139c17c3e7a57482be70eabc8e87 not found: ID does not exist"
Jul 17 13:56:11.510360 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:56:11.510329    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" is forbidden: User \"system:node:ip-10-0-129-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-191.ec2.internal' and this object" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:12.072989 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:56:12.072952    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-v5wmr\" is forbidden: User \"system:node:ip-10-0-129-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-191.ec2.internal' and this object" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v5wmr"
Jul 17 13:56:12.074870 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:12.074843    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" path="/var/lib/kubelet/pods/7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24/volumes"
Jul 17 13:56:21.502408 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:21.502380    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:34.585405 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:34.585373    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"]
Jul 17 13:56:34.585751 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:34.585576    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6" podUID="b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" containerName="manager" containerID="cri-o://b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f" gracePeriod=10
Jul 17 13:56:34.824225 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:34.824205    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:35.013797 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.013772    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-extensions-socket-volume\") pod \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") "
Jul 17 13:56:35.013964 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.013839    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6w5s\" (UniqueName: \"kubernetes.io/projected/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-kube-api-access-c6w5s\") pod \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\" (UID: \"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5\") "
Jul 17 13:56:35.014163 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.014139    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" (UID: "b9a043e5-99ce-48d3-8ddb-7b4df2308ec5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:56:35.015797 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.015775    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-kube-api-access-c6w5s" (OuterVolumeSpecName: "kube-api-access-c6w5s") pod "b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" (UID: "b9a043e5-99ce-48d3-8ddb-7b4df2308ec5"). InnerVolumeSpecName "kube-api-access-c6w5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:56:35.114643 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.114621    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-extensions-socket-volume\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:56:35.114643 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.114641    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6w5s\" (UniqueName: \"kubernetes.io/projected/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5-kube-api-access-c6w5s\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:56:35.613564 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.613524    2575 generic.go:356] "Generic (PLEG): container finished" podID="b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" containerID="b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f" exitCode=0
Jul 17 13:56:35.613951 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.613594    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"
Jul 17 13:56:35.613951 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.613618    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6" event={"ID":"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5","Type":"ContainerDied","Data":"b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f"}
Jul 17 13:56:35.613951 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.613703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6" event={"ID":"b9a043e5-99ce-48d3-8ddb-7b4df2308ec5","Type":"ContainerDied","Data":"668f1508ec42ff61d12a78e3eb8e150988a3cafd2a8b4162d9a5d4c0e14ef2ef"}
Jul 17 13:56:35.613951 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.613729    2575 scope.go:117] "RemoveContainer" containerID="b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f"
Jul 17 13:56:35.623804 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.623783    2575 scope.go:117] "RemoveContainer" containerID="b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f"
Jul 17 13:56:35.624061 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:56:35.624040    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f\": container with ID starting with b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f not found: ID does not exist" containerID="b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f"
Jul 17 13:56:35.624130 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.624074    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f"} err="failed to get container status \"b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f\": rpc error: code = NotFound desc = could not find container \"b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f\": container with ID starting with b873f1f4c4c56acaa6de3330c1e672078c5a4a3fc68acd60add46331dbcbc84f not found: ID does not exist"
Jul 17 13:56:35.636763 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.636737    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"]
Jul 17 13:56:35.642819 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:35.642796    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gmgx6"]
Jul 17 13:56:36.074244 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:36.074215    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" path="/var/lib/kubelet/pods/b9a043e5-99ce-48d3-8ddb-7b4df2308ec5/volumes"
Jul 17 13:56:50.506116 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.506077    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p9p22"]
Jul 17 13:56:50.506655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.506468    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" containerName="manager"
Jul 17 13:56:50.506655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.506484    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" containerName="manager"
Jul 17 13:56:50.506655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.506512    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" containerName="manager"
Jul 17 13:56:50.506655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.506518    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" containerName="manager"
Jul 17 13:56:50.506655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.506572    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd5b617-ddc5-4d79-8d5a-7d5d39c7de24" containerName="manager"
Jul 17 13:56:50.506655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.506584    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a043e5-99ce-48d3-8ddb-7b4df2308ec5" containerName="manager"
Jul 17 13:56:50.508565 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.508546    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.511044 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.511017    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 13:56:50.511217 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.511039    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 13:56:50.511777 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.511756    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 13:56:50.511858 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.511819    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gvwvs\""
Jul 17 13:56:50.521654 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.521621    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p9p22"]
Jul 17 13:56:50.528159 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.528131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/990361aa-a92d-4b98-8ffd-ff41836f8f81-config-file\") pod \"limitador-limitador-64c94fd64f-p9p22\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.528307 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.528173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqtnv\" (UniqueName: \"kubernetes.io/projected/990361aa-a92d-4b98-8ffd-ff41836f8f81-kube-api-access-fqtnv\") pod \"limitador-limitador-64c94fd64f-p9p22\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.604652 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.604616    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p9p22"]
Jul 17 13:56:50.629323 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.629302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/990361aa-a92d-4b98-8ffd-ff41836f8f81-config-file\") pod \"limitador-limitador-64c94fd64f-p9p22\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.629438 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.629338    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqtnv\" (UniqueName: \"kubernetes.io/projected/990361aa-a92d-4b98-8ffd-ff41836f8f81-kube-api-access-fqtnv\") pod \"limitador-limitador-64c94fd64f-p9p22\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.629891 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.629870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/990361aa-a92d-4b98-8ffd-ff41836f8f81-config-file\") pod \"limitador-limitador-64c94fd64f-p9p22\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.638086 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.638064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqtnv\" (UniqueName: \"kubernetes.io/projected/990361aa-a92d-4b98-8ffd-ff41836f8f81-kube-api-access-fqtnv\") pod \"limitador-limitador-64c94fd64f-p9p22\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.820174 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.820100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:50.950324 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:50.950300    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p9p22"]
Jul 17 13:56:50.952388 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:56:50.952362    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990361aa_a92d_4b98_8ffd_ff41836f8f81.slice/crio-38c220272bfea8d177a644b49e413ed9b4470e8680cdb41feab46501880a2dba WatchSource:0}: Error finding container 38c220272bfea8d177a644b49e413ed9b4470e8680cdb41feab46501880a2dba: Status 404 returned error can't find the container with id 38c220272bfea8d177a644b49e413ed9b4470e8680cdb41feab46501880a2dba
Jul 17 13:56:51.690788 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:51.690763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22" event={"ID":"990361aa-a92d-4b98-8ffd-ff41836f8f81","Type":"ContainerStarted","Data":"38c220272bfea8d177a644b49e413ed9b4470e8680cdb41feab46501880a2dba"}
Jul 17 13:56:53.701259 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:53.701229    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22" event={"ID":"990361aa-a92d-4b98-8ffd-ff41836f8f81","Type":"ContainerStarted","Data":"7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55"}
Jul 17 13:56:53.701592 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:53.701350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:56:53.717790 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:56:53.717737    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22" podStartSLOduration=1.101736363 podStartE2EDuration="3.717721103s" totalImagesPullingTime="2.61598474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:56:53.716096246 +0000 UTC m=+692.103443680" watchObservedRunningTime="2026-07-17 13:56:53.717721103 +0000 UTC m=+692.105068542"
Jul 17 13:57:04.709227 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:04.709195    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:57:05.374616 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:05.374579    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p9p22"]
Jul 17 13:57:05.374805 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:05.374749    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22" podUID="990361aa-a92d-4b98-8ffd-ff41836f8f81" containerName="limitador" containerID="cri-o://7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55" gracePeriod=30
Jul 17 13:57:05.909510 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:05.909487    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:57:06.048013 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.047990    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqtnv\" (UniqueName: \"kubernetes.io/projected/990361aa-a92d-4b98-8ffd-ff41836f8f81-kube-api-access-fqtnv\") pod \"990361aa-a92d-4b98-8ffd-ff41836f8f81\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") "
Jul 17 13:57:06.048150 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.048026    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/990361aa-a92d-4b98-8ffd-ff41836f8f81-config-file\") pod \"990361aa-a92d-4b98-8ffd-ff41836f8f81\" (UID: \"990361aa-a92d-4b98-8ffd-ff41836f8f81\") "
Jul 17 13:57:06.048433 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.048413    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990361aa-a92d-4b98-8ffd-ff41836f8f81-config-file" (OuterVolumeSpecName: "config-file") pod "990361aa-a92d-4b98-8ffd-ff41836f8f81" (UID: "990361aa-a92d-4b98-8ffd-ff41836f8f81"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:57:06.049930 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.049908    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990361aa-a92d-4b98-8ffd-ff41836f8f81-kube-api-access-fqtnv" (OuterVolumeSpecName: "kube-api-access-fqtnv") pod "990361aa-a92d-4b98-8ffd-ff41836f8f81" (UID: "990361aa-a92d-4b98-8ffd-ff41836f8f81"). InnerVolumeSpecName "kube-api-access-fqtnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:57:06.149092 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.149064    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqtnv\" (UniqueName: \"kubernetes.io/projected/990361aa-a92d-4b98-8ffd-ff41836f8f81-kube-api-access-fqtnv\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:57:06.149092 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.149091    2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/990361aa-a92d-4b98-8ffd-ff41836f8f81-config-file\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:57:06.769069 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.769033    2575 generic.go:356] "Generic (PLEG): container finished" podID="990361aa-a92d-4b98-8ffd-ff41836f8f81" containerID="7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55" exitCode=0
Jul 17 13:57:06.769207 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.769083    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22"
Jul 17 13:57:06.769207 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.769102    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22" event={"ID":"990361aa-a92d-4b98-8ffd-ff41836f8f81","Type":"ContainerDied","Data":"7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55"}
Jul 17 13:57:06.769207 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.769153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p9p22" event={"ID":"990361aa-a92d-4b98-8ffd-ff41836f8f81","Type":"ContainerDied","Data":"38c220272bfea8d177a644b49e413ed9b4470e8680cdb41feab46501880a2dba"}
Jul 17 13:57:06.769207 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.769179    2575 scope.go:117] "RemoveContainer" containerID="7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55"
Jul 17 13:57:06.778758 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.778740    2575 scope.go:117] "RemoveContainer" containerID="7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55"
Jul 17 13:57:06.779054 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:57:06.779032    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55\": container with ID starting with 7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55 not found: ID does not exist" containerID="7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55"
Jul 17 13:57:06.779114 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.779063    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55"} err="failed to get container status \"7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55\": rpc error: code = NotFound desc = could not find container \"7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55\": container with ID starting with 7cf0e4143c18780c5580bde1ffe1617b9a11215f13e17300219acae47f0bdc55 not found: ID does not exist"
Jul 17 13:57:06.785028 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.784984    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p9p22"]
Jul 17 13:57:06.788601 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:06.788575    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p9p22"]
Jul 17 13:57:07.481406 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.481377    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"]
Jul 17 13:57:07.481910 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.481894    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990361aa-a92d-4b98-8ffd-ff41836f8f81" containerName="limitador"
Jul 17 13:57:07.481955 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.481913    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="990361aa-a92d-4b98-8ffd-ff41836f8f81" containerName="limitador"
Jul 17 13:57:07.482017 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.482005    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="990361aa-a92d-4b98-8ffd-ff41836f8f81" containerName="limitador"
Jul 17 13:57:07.485012 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.484994    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.487518 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.487494    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:57:07.487518 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.487511    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 13:57:07.487701 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.487515    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:57:07.488359 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.488344    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mtds6\""
Jul 17 13:57:07.494645 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.494622    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"]
Jul 17 13:57:07.559833 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.559810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngs9m\" (UniqueName: \"kubernetes.io/projected/d6050374-321f-48bb-aa8f-3d6040368eea-kube-api-access-ngs9m\") pod \"postgres-69fb4594cf-z2njc\" (UID: \"d6050374-321f-48bb-aa8f-3d6040368eea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.559934 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.559855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6050374-321f-48bb-aa8f-3d6040368eea-data\") pod \"postgres-69fb4594cf-z2njc\" (UID: \"d6050374-321f-48bb-aa8f-3d6040368eea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.660546 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.660524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6050374-321f-48bb-aa8f-3d6040368eea-data\") pod \"postgres-69fb4594cf-z2njc\" (UID: \"d6050374-321f-48bb-aa8f-3d6040368eea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.660650 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.660584    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngs9m\" (UniqueName: \"kubernetes.io/projected/d6050374-321f-48bb-aa8f-3d6040368eea-kube-api-access-ngs9m\") pod \"postgres-69fb4594cf-z2njc\" (UID: \"d6050374-321f-48bb-aa8f-3d6040368eea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.660933 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.660913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6050374-321f-48bb-aa8f-3d6040368eea-data\") pod \"postgres-69fb4594cf-z2njc\" (UID: \"d6050374-321f-48bb-aa8f-3d6040368eea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.674193 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.674165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngs9m\" (UniqueName: \"kubernetes.io/projected/d6050374-321f-48bb-aa8f-3d6040368eea-kube-api-access-ngs9m\") pod \"postgres-69fb4594cf-z2njc\" (UID: \"d6050374-321f-48bb-aa8f-3d6040368eea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.795128 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.795056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:07.932809 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:07.932751    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"]
Jul 17 13:57:07.935054 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:57:07.935024    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6050374_321f_48bb_aa8f_3d6040368eea.slice/crio-56921b0dcf12a5e5da64ae0b00a1e282a67fd249daa07fae0ff1cacf8652acff WatchSource:0}: Error finding container 56921b0dcf12a5e5da64ae0b00a1e282a67fd249daa07fae0ff1cacf8652acff: Status 404 returned error can't find the container with id 56921b0dcf12a5e5da64ae0b00a1e282a67fd249daa07fae0ff1cacf8652acff
Jul 17 13:57:08.074710 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:08.074645    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="990361aa-a92d-4b98-8ffd-ff41836f8f81" path="/var/lib/kubelet/pods/990361aa-a92d-4b98-8ffd-ff41836f8f81/volumes"
Jul 17 13:57:08.781167 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:08.781134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc" event={"ID":"d6050374-321f-48bb-aa8f-3d6040368eea","Type":"ContainerStarted","Data":"56921b0dcf12a5e5da64ae0b00a1e282a67fd249daa07fae0ff1cacf8652acff"}
Jul 17 13:57:13.806648 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:13.806613    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc" event={"ID":"d6050374-321f-48bb-aa8f-3d6040368eea","Type":"ContainerStarted","Data":"e6d97646b5255bd1f5739177989113033ff526561c3ac87f142ee773cc2e342c"}
Jul 17 13:57:13.807004 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:13.806674    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:13.823182 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:13.823140    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc" podStartSLOduration=1.908196635 podStartE2EDuration="6.823125632s" totalImagesPullingTime="4.914928997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:57:13.821888727 +0000 UTC m=+712.209236166" watchObservedRunningTime="2026-07-17 13:57:13.823125632 +0000 UTC m=+712.210473071"
Jul 17 13:57:19.840490 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:19.840413    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z2njc"
Jul 17 13:57:27.553460 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.553429    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c5c984cc-r5ht7"]
Jul 17 13:57:27.570189 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.570159    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.572676 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.572652    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c5c984cc-r5ht7"]
Jul 17 13:57:27.573243 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.573229    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xtg2f\""
Jul 17 13:57:27.573347 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.573229    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 13:57:27.573715 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.573698    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 13:57:27.713093 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.713070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5200ee1-e247-417a-bdb2-5049ecc48e88-webhook-cert\") pod \"maas-controller-68c5c984cc-r5ht7\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") " pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.713197 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.713130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tl5\" (UniqueName: \"kubernetes.io/projected/c5200ee1-e247-417a-bdb2-5049ecc48e88-kube-api-access-d4tl5\") pod \"maas-controller-68c5c984cc-r5ht7\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") " pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.751143 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.751119    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f484f85fb-2stg5"]
Jul 17 13:57:27.753486 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.753469    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:27.771208 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.771182    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f484f85fb-2stg5"]
Jul 17 13:57:27.813733 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.813686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tl5\" (UniqueName: \"kubernetes.io/projected/c5200ee1-e247-417a-bdb2-5049ecc48e88-kube-api-access-d4tl5\") pod \"maas-controller-68c5c984cc-r5ht7\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") " pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.813733 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.813724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5200ee1-e247-417a-bdb2-5049ecc48e88-webhook-cert\") pod \"maas-controller-68c5c984cc-r5ht7\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") " pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.816015 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.815998    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5200ee1-e247-417a-bdb2-5049ecc48e88-webhook-cert\") pod \"maas-controller-68c5c984cc-r5ht7\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") " pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.822748 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.822729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tl5\" (UniqueName: \"kubernetes.io/projected/c5200ee1-e247-417a-bdb2-5049ecc48e88-kube-api-access-d4tl5\") pod \"maas-controller-68c5c984cc-r5ht7\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") " pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.881467 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.881449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:27.914385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.914359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec4a5837-9d40-488d-9753-f8b584f9bc3e-webhook-cert\") pod \"maas-controller-f484f85fb-2stg5\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") " pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:27.914497 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:27.914439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk8bm\" (UniqueName: \"kubernetes.io/projected/ec4a5837-9d40-488d-9753-f8b584f9bc3e-kube-api-access-wk8bm\") pod \"maas-controller-f484f85fb-2stg5\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") " pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:28.004440 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.004415    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c5c984cc-r5ht7"]
Jul 17 13:57:28.006857 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:57:28.006832    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5200ee1_e247_417a_bdb2_5049ecc48e88.slice/crio-3deb4d8fe010efc359f09490ec8dbb202e09b5689f1bdf84521cc69189d8a5ba WatchSource:0}: Error finding container 3deb4d8fe010efc359f09490ec8dbb202e09b5689f1bdf84521cc69189d8a5ba: Status 404 returned error can't find the container with id 3deb4d8fe010efc359f09490ec8dbb202e09b5689f1bdf84521cc69189d8a5ba
Jul 17 13:57:28.015025 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.015003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk8bm\" (UniqueName: \"kubernetes.io/projected/ec4a5837-9d40-488d-9753-f8b584f9bc3e-kube-api-access-wk8bm\") pod \"maas-controller-f484f85fb-2stg5\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") " pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:28.015109 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.015035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec4a5837-9d40-488d-9753-f8b584f9bc3e-webhook-cert\") pod \"maas-controller-f484f85fb-2stg5\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") " pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:28.017189 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.017167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec4a5837-9d40-488d-9753-f8b584f9bc3e-webhook-cert\") pod \"maas-controller-f484f85fb-2stg5\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") " pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:28.025799 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.025782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk8bm\" (UniqueName: \"kubernetes.io/projected/ec4a5837-9d40-488d-9753-f8b584f9bc3e-kube-api-access-wk8bm\") pod \"maas-controller-f484f85fb-2stg5\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") " pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:28.062798 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.062779    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:28.184021 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.183998    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f484f85fb-2stg5"]
Jul 17 13:57:28.185516 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:57:28.185490    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4a5837_9d40_488d_9753_f8b584f9bc3e.slice/crio-1abd9b2d03ed37d0b831c4e2a8554207853c6d7e123c3d5c995fb5925a20a176 WatchSource:0}: Error finding container 1abd9b2d03ed37d0b831c4e2a8554207853c6d7e123c3d5c995fb5925a20a176: Status 404 returned error can't find the container with id 1abd9b2d03ed37d0b831c4e2a8554207853c6d7e123c3d5c995fb5925a20a176
Jul 17 13:57:28.887840 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.887801    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-2stg5" event={"ID":"ec4a5837-9d40-488d-9753-f8b584f9bc3e","Type":"ContainerStarted","Data":"1abd9b2d03ed37d0b831c4e2a8554207853c6d7e123c3d5c995fb5925a20a176"}
Jul 17 13:57:28.890979 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:28.890944    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c5c984cc-r5ht7" event={"ID":"c5200ee1-e247-417a-bdb2-5049ecc48e88","Type":"ContainerStarted","Data":"3deb4d8fe010efc359f09490ec8dbb202e09b5689f1bdf84521cc69189d8a5ba"}
Jul 17 13:57:30.902408 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:30.902374    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-2stg5" event={"ID":"ec4a5837-9d40-488d-9753-f8b584f9bc3e","Type":"ContainerStarted","Data":"f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6"}
Jul 17 13:57:30.902826 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:30.902442    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:30.904057 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:30.904030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c5c984cc-r5ht7" event={"ID":"c5200ee1-e247-417a-bdb2-5049ecc48e88","Type":"ContainerStarted","Data":"7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120"}
Jul 17 13:57:30.904166 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:30.904073    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:31.050521 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:31.050473    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f484f85fb-2stg5" podStartSLOduration=1.549299257 podStartE2EDuration="4.050455479s" totalImagesPullingTime="2.501156222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:57:30.998804364 +0000 UTC m=+729.386151839" watchObservedRunningTime="2026-07-17 13:57:31.050455479 +0000 UTC m=+729.437802919"
Jul 17 13:57:31.051533 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:31.051503    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c5c984cc-r5ht7" podStartSLOduration=1.377974005 podStartE2EDuration="4.051492419s" totalImagesPullingTime="2.673518414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:57:31.050580745 +0000 UTC m=+729.437928208" watchObservedRunningTime="2026-07-17 13:57:31.051492419 +0000 UTC m=+729.438839858"
Jul 17 13:57:41.913752 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:41.913718    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 13:57:41.914882 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:41.914855    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:41.978516 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:41.978479    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c5c984cc-r5ht7"]
Jul 17 13:57:41.978694 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:41.978657    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68c5c984cc-r5ht7" podUID="c5200ee1-e247-417a-bdb2-5049ecc48e88" containerName="manager" containerID="cri-o://7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120" gracePeriod=30
Jul 17 13:57:42.214195 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.214167    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:42.223646 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.223618    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4tl5\" (UniqueName: \"kubernetes.io/projected/c5200ee1-e247-417a-bdb2-5049ecc48e88-kube-api-access-d4tl5\") pod \"c5200ee1-e247-417a-bdb2-5049ecc48e88\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") "
Jul 17 13:57:42.223754 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.223687    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5200ee1-e247-417a-bdb2-5049ecc48e88-webhook-cert\") pod \"c5200ee1-e247-417a-bdb2-5049ecc48e88\" (UID: \"c5200ee1-e247-417a-bdb2-5049ecc48e88\") "
Jul 17 13:57:42.225639 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.225615    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5200ee1-e247-417a-bdb2-5049ecc48e88-kube-api-access-d4tl5" (OuterVolumeSpecName: "kube-api-access-d4tl5") pod "c5200ee1-e247-417a-bdb2-5049ecc48e88" (UID: "c5200ee1-e247-417a-bdb2-5049ecc48e88"). InnerVolumeSpecName "kube-api-access-d4tl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:57:42.225740 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.225712    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5200ee1-e247-417a-bdb2-5049ecc48e88-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c5200ee1-e247-417a-bdb2-5049ecc48e88" (UID: "c5200ee1-e247-417a-bdb2-5049ecc48e88"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:57:42.324486 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.324460    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4tl5\" (UniqueName: \"kubernetes.io/projected/c5200ee1-e247-417a-bdb2-5049ecc48e88-kube-api-access-d4tl5\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:57:42.324486 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.324483    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5200ee1-e247-417a-bdb2-5049ecc48e88-webhook-cert\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 13:57:42.967654 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.967620    2575 generic.go:356] "Generic (PLEG): container finished" podID="c5200ee1-e247-417a-bdb2-5049ecc48e88" containerID="7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120" exitCode=0
Jul 17 13:57:42.968086 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.967676    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c5c984cc-r5ht7"
Jul 17 13:57:42.968086 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.967710    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c5c984cc-r5ht7" event={"ID":"c5200ee1-e247-417a-bdb2-5049ecc48e88","Type":"ContainerDied","Data":"7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120"}
Jul 17 13:57:42.968086 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.967774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c5c984cc-r5ht7" event={"ID":"c5200ee1-e247-417a-bdb2-5049ecc48e88","Type":"ContainerDied","Data":"3deb4d8fe010efc359f09490ec8dbb202e09b5689f1bdf84521cc69189d8a5ba"}
Jul 17 13:57:42.968086 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.967806    2575 scope.go:117] "RemoveContainer" containerID="7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120"
Jul 17 13:57:42.977471 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.977452    2575 scope.go:117] "RemoveContainer" containerID="7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120"
Jul 17 13:57:42.977771 ip-10-0-129-191 kubenswrapper[2575]: E0717 13:57:42.977754    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120\": container with ID starting with 7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120 not found: ID does not exist" containerID="7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120"
Jul 17 13:57:42.977810 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.977779    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120"} err="failed to get container status \"7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120\": rpc error: code = NotFound desc = could not find container \"7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120\": container with ID starting with 7b4a5bf810c48bb862cf44e1d7eb5e99bfd65af1b1050617e04b30e95b374120 not found: ID does not exist"
Jul 17 13:57:42.990529 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.990498    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c5c984cc-r5ht7"]
Jul 17 13:57:42.996186 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:42.996161    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c5c984cc-r5ht7"]
Jul 17 13:57:44.075172 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:57:44.075134    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5200ee1-e247-417a-bdb2-5049ecc48e88" path="/var/lib/kubelet/pods/c5200ee1-e247-417a-bdb2-5049ecc48e88/volumes"
Jul 17 13:58:04.484254 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.484221    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75c9748d6-j459j"]
Jul 17 13:58:04.484718 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.484585    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5200ee1-e247-417a-bdb2-5049ecc48e88" containerName="manager"
Jul 17 13:58:04.484718 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.484598    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5200ee1-e247-417a-bdb2-5049ecc48e88" containerName="manager"
Jul 17 13:58:04.484718 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.484663    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5200ee1-e247-417a-bdb2-5049ecc48e88" containerName="manager"
Jul 17 13:58:04.489030 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.489008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.491317 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.491295    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 17 13:58:04.491441 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.491352    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgjc5\""
Jul 17 13:58:04.498202 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.498178    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75c9748d6-j459j"]
Jul 17 13:58:04.588536 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.588509    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7c627d-7071-4c42-beb8-8ed0acae9279-maas-api-tls\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.588642 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.588542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c627d-7071-4c42-beb8-8ed0acae9279-openshift-service-ca\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.588642 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.588573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wcm\" (UniqueName: \"kubernetes.io/projected/ac7c627d-7071-4c42-beb8-8ed0acae9279-kube-api-access-j2wcm\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.689891 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.689861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7c627d-7071-4c42-beb8-8ed0acae9279-maas-api-tls\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.689891 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.689894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c627d-7071-4c42-beb8-8ed0acae9279-openshift-service-ca\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.690025 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.689931    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wcm\" (UniqueName: \"kubernetes.io/projected/ac7c627d-7071-4c42-beb8-8ed0acae9279-kube-api-access-j2wcm\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.690550 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.690528    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c627d-7071-4c42-beb8-8ed0acae9279-openshift-service-ca\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.692077 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.692051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7c627d-7071-4c42-beb8-8ed0acae9279-maas-api-tls\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.698050 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.698028    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wcm\" (UniqueName: \"kubernetes.io/projected/ac7c627d-7071-4c42-beb8-8ed0acae9279-kube-api-access-j2wcm\") pod \"maas-api-75c9748d6-j459j\" (UID: \"ac7c627d-7071-4c42-beb8-8ed0acae9279\") " pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.799036 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.798972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:04.930219 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:04.930101    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75c9748d6-j459j"]
Jul 17 13:58:04.932482 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:58:04.932452    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7c627d_7071_4c42_beb8_8ed0acae9279.slice/crio-629708152b1e3d6ae0166c96d47cd0a5b2fc60c203fc819fafb28d0d870cc5c5 WatchSource:0}: Error finding container 629708152b1e3d6ae0166c96d47cd0a5b2fc60c203fc819fafb28d0d870cc5c5: Status 404 returned error can't find the container with id 629708152b1e3d6ae0166c96d47cd0a5b2fc60c203fc819fafb28d0d870cc5c5
Jul 17 13:58:05.064619 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.064554    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"]
Jul 17 13:58:05.068263 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.068241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.070559 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.070538    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 17 13:58:05.070655 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.070616    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5ghpv\""
Jul 17 13:58:05.074562 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.074541    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j" event={"ID":"ac7c627d-7071-4c42-beb8-8ed0acae9279","Type":"ContainerStarted","Data":"629708152b1e3d6ae0166c96d47cd0a5b2fc60c203fc819fafb28d0d870cc5c5"}
Jul 17 13:58:05.078802 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.078781    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"]
Jul 17 13:58:05.090606 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.090577    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-j8rdn"]
Jul 17 13:58:05.094027 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.094008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.104186 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.104160    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-j8rdn"]
Jul 17 13:58:05.195583 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.195563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55924f23-340f-407d-824d-ddff617a1f7a-plugins-config-volume\") pod \"payload-processing-5bf47cd988-j8rdn\" (UID: \"55924f23-340f-407d-824d-ddff617a1f7a\") " pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.195683 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.195615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cwg\" (UniqueName: \"kubernetes.io/projected/0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da-kube-api-access-95cwg\") pod \"payload-pre-processing-7677b774bb-lm4zz\" (UID: \"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.195729 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.195699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h426x\" (UniqueName: \"kubernetes.io/projected/55924f23-340f-407d-824d-ddff617a1f7a-kube-api-access-h426x\") pod \"payload-processing-5bf47cd988-j8rdn\" (UID: \"55924f23-340f-407d-824d-ddff617a1f7a\") " pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.195729 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.195721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-lm4zz\" (UID: \"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.296378 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.296358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h426x\" (UniqueName: \"kubernetes.io/projected/55924f23-340f-407d-824d-ddff617a1f7a-kube-api-access-h426x\") pod \"payload-processing-5bf47cd988-j8rdn\" (UID: \"55924f23-340f-407d-824d-ddff617a1f7a\") " pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.296509 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.296386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-lm4zz\" (UID: \"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.296509 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.296415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55924f23-340f-407d-824d-ddff617a1f7a-plugins-config-volume\") pod \"payload-processing-5bf47cd988-j8rdn\" (UID: \"55924f23-340f-407d-824d-ddff617a1f7a\") " pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.296509 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.296466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cwg\" (UniqueName: \"kubernetes.io/projected/0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da-kube-api-access-95cwg\") pod \"payload-pre-processing-7677b774bb-lm4zz\" (UID: \"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.297108 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.297088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-lm4zz\" (UID: \"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.297145 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.297125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55924f23-340f-407d-824d-ddff617a1f7a-plugins-config-volume\") pod \"payload-processing-5bf47cd988-j8rdn\" (UID: \"55924f23-340f-407d-824d-ddff617a1f7a\") " pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.304772 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.304747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cwg\" (UniqueName: \"kubernetes.io/projected/0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da-kube-api-access-95cwg\") pod \"payload-pre-processing-7677b774bb-lm4zz\" (UID: \"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.304772 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.304764    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h426x\" (UniqueName: \"kubernetes.io/projected/55924f23-340f-407d-824d-ddff617a1f7a-kube-api-access-h426x\") pod \"payload-processing-5bf47cd988-j8rdn\" (UID: \"55924f23-340f-407d-824d-ddff617a1f7a\") " pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.379359 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.379305    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:05.404124 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.404096    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:05.516779 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.516723    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"]
Jul 17 13:58:05.519723 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:58:05.519660    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c19cd24_ad9e_4e8c_9d86_c85a7d1a75da.slice/crio-ed0e31bd047546dd69a2515969244177fa68f462f74c38d39dea288c5c3df16e WatchSource:0}: Error finding container ed0e31bd047546dd69a2515969244177fa68f462f74c38d39dea288c5c3df16e: Status 404 returned error can't find the container with id ed0e31bd047546dd69a2515969244177fa68f462f74c38d39dea288c5c3df16e
Jul 17 13:58:05.547876 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:05.547853    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-j8rdn"]
Jul 17 13:58:05.550305 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:58:05.550270    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55924f23_340f_407d_824d_ddff617a1f7a.slice/crio-bbb826f6791913a6a158618fdacfc27b1dbe92dfcb732d7e68a428df1aefb739 WatchSource:0}: Error finding container bbb826f6791913a6a158618fdacfc27b1dbe92dfcb732d7e68a428df1aefb739: Status 404 returned error can't find the container with id bbb826f6791913a6a158618fdacfc27b1dbe92dfcb732d7e68a428df1aefb739
Jul 17 13:58:06.086392 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:06.085987    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz" event={"ID":"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da","Type":"ContainerStarted","Data":"ed0e31bd047546dd69a2515969244177fa68f462f74c38d39dea288c5c3df16e"}
Jul 17 13:58:06.087981 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:06.087875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn" event={"ID":"55924f23-340f-407d-824d-ddff617a1f7a","Type":"ContainerStarted","Data":"bbb826f6791913a6a158618fdacfc27b1dbe92dfcb732d7e68a428df1aefb739"}
Jul 17 13:58:07.101909 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:07.101827    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j" event={"ID":"ac7c627d-7071-4c42-beb8-8ed0acae9279","Type":"ContainerStarted","Data":"d4f9d875209377f6876ec92c4487168ac677a752077f0e6a8598853549482b7c"}
Jul 17 13:58:07.101909 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:07.101928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:07.126114 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:07.126057    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j" podStartSLOduration=1.25560843 podStartE2EDuration="3.126037831s" totalImagesPullingTime="1.870429401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:58:07.124235937 +0000 UTC m=+765.511583380" watchObservedRunningTime="2026-07-17 13:58:07.126037831 +0000 UTC m=+765.513385273"
Jul 17 13:58:08.108027 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:08.107991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn" event={"ID":"55924f23-340f-407d-824d-ddff617a1f7a","Type":"ContainerStarted","Data":"da4c813d169bf29b9d87f6f3383a4eae61bb895b3b10c7bb33934a7c042d8fa0"}
Jul 17 13:58:08.108489 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:08.108049    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:08.109595 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:08.109573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz" event={"ID":"0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da","Type":"ContainerStarted","Data":"405fca7716f5f15f2cb5c0f0c401e1c94f14fa656e186806acc10d3e2cf266b9"}
Jul 17 13:58:08.109725 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:08.109611    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:08.125707 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:08.125654    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn" podStartSLOduration=1.373503273 podStartE2EDuration="3.125636623s" totalImagesPullingTime="1.75213335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:58:08.124832957 +0000 UTC m=+766.512180397" watchObservedRunningTime="2026-07-17 13:58:08.125636623 +0000 UTC m=+766.512984064"
Jul 17 13:58:08.140801 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:08.140765    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz" podStartSLOduration=1.352789994 podStartE2EDuration="3.140752435s" totalImagesPullingTime="1.787962441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:58:08.140378589 +0000 UTC m=+766.527726028" watchObservedRunningTime="2026-07-17 13:58:08.140752435 +0000 UTC m=+766.528099927"
Jul 17 13:58:13.115238 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:13.115206    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75c9748d6-j459j"
Jul 17 13:58:19.116151 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:19.116123    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-j8rdn"
Jul 17 13:58:19.116502 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:19.116305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-lm4zz"
Jul 17 13:58:32.175251 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.175211    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"]
Jul 17 13:58:32.179039 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.179011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.181518 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.181482    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 13:58:32.181648 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.181607    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 17 13:58:32.181718 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.181663    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 13:58:32.182344 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.182325    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wn44r\""
Jul 17 13:58:32.186297 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.186252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"]
Jul 17 13:58:32.295049 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.295025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxn5\" (UniqueName: \"kubernetes.io/projected/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-kube-api-access-drxn5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.295166 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.295078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.295166 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.295135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.295252 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.295167    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.295252 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.295189    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.295252 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.295215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.396436 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.396531 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396445    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.396531 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.396531 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxn5\" (UniqueName: \"kubernetes.io/projected/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-kube-api-access-drxn5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.396662 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.396662 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.396944 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396916    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.397018 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396923    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.397018 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.396978    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.398694 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.398674    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.398936 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.398919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.404395 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.404377    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxn5\" (UniqueName: \"kubernetes.io/projected/ec71b2e3-2930-4625-b4f3-9572a0f9ccbf-kube-api-access-drxn5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q\" (UID: \"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.491817 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.491798    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:32.618907 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:32.618870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"]
Jul 17 13:58:32.622943 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:58:32.622918    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec71b2e3_2930_4625_b4f3_9572a0f9ccbf.slice/crio-a06c7cb49158aabd725fb8a78347b5a15241b0e12eadd2617d31d111195499ba WatchSource:0}: Error finding container a06c7cb49158aabd725fb8a78347b5a15241b0e12eadd2617d31d111195499ba: Status 404 returned error can't find the container with id a06c7cb49158aabd725fb8a78347b5a15241b0e12eadd2617d31d111195499ba
Jul 17 13:58:33.233347 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:33.233311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q" event={"ID":"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf","Type":"ContainerStarted","Data":"a06c7cb49158aabd725fb8a78347b5a15241b0e12eadd2617d31d111195499ba"}
Jul 17 13:58:34.239897 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:34.239862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q" event={"ID":"ec71b2e3-2930-4625-b4f3-9572a0f9ccbf","Type":"ContainerStarted","Data":"f59e2cd5b7631191658d1627f93be2d63da3bc97dde677530efd405430f4fd6b"}
Jul 17 13:58:34.261409 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:34.261357    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q" podStartSLOduration=1.233557886 podStartE2EDuration="2.261339803s" totalImagesPullingTime="1.027781917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:58:34.260044016 +0000 UTC m=+792.647391457" watchObservedRunningTime="2026-07-17 13:58:34.261339803 +0000 UTC m=+792.648687242"
Jul 17 13:58:42.491983 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:42.491937    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:42.492422 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:42.492021    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:42.504315 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:42.504293    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:58:43.294886 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:58:43.294852    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q"
Jul 17 13:59:01.272006 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.271971    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"]
Jul 17 13:59:01.275514 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.275492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.277972 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.277954    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 17 13:59:01.286392 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.286372    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"]
Jul 17 13:59:01.419383 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.419354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.419509 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.419394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.419509 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.419414    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.419624 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.419523    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.419624 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.419584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.419707 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.419619    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrws\" (UniqueName: \"kubernetes.io/projected/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-kube-api-access-jzrws\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520181 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520313 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520313 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrws\" (UniqueName: \"kubernetes.io/projected/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-kube-api-access-jzrws\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520313 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520478 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520478 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520589 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520541    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520681 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.520720 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.520710    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.522602 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.522555    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.522674 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.522657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.528204 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.528181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrws\" (UniqueName: \"kubernetes.io/projected/bc89dbee-026b-4a0f-a3d8-3eb6a652f32d-kube-api-access-jzrws\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdrb8\" (UID: \"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.586076 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.586058    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:01.711219 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:01.711189    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"]
Jul 17 13:59:01.714596 ip-10-0-129-191 kubenswrapper[2575]: W0717 13:59:01.714568    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc89dbee_026b_4a0f_a3d8_3eb6a652f32d.slice/crio-9c574c01602b87b0d2eadd53e593b2774a6fc324905b7891196c6225ded5c07e WatchSource:0}: Error finding container 9c574c01602b87b0d2eadd53e593b2774a6fc324905b7891196c6225ded5c07e: Status 404 returned error can't find the container with id 9c574c01602b87b0d2eadd53e593b2774a6fc324905b7891196c6225ded5c07e
Jul 17 13:59:02.371504 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:02.371430    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8" event={"ID":"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d","Type":"ContainerStarted","Data":"9ff0695c0344e31cc22c69a5ea3ca184f3644264c0e2e902ef1bd1a0ff850624"}
Jul 17 13:59:02.371504 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:02.371479    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8" event={"ID":"bc89dbee-026b-4a0f-a3d8-3eb6a652f32d","Type":"ContainerStarted","Data":"9c574c01602b87b0d2eadd53e593b2774a6fc324905b7891196c6225ded5c07e"}
Jul 17 13:59:02.391385 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:02.391070    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8" podStartSLOduration=1.099598347 podStartE2EDuration="1.391052249s" totalImagesPullingTime="291.453902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:59:02.388393731 +0000 UTC m=+820.775741172" watchObservedRunningTime="2026-07-17 13:59:02.391052249 +0000 UTC m=+820.778399690"
Jul 17 13:59:11.586642 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:11.586606    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:11.587065 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:11.586676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:11.599414 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:11.599384    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 13:59:12.446555 ip-10-0-129-191 kubenswrapper[2575]: I0717 13:59:12.446527    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdrb8"
Jul 17 14:00:22.007130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:00:22.007102    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:00:22.010839 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:00:22.010814    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:00:22.011494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:00:22.011474    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:00:22.014634 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:00:22.014615    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:00:22.018318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:00:22.018300    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:00:22.021354 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:00:22.021337    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:01:06.976207 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:06.976165    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f484f85fb-2stg5"]
Jul 17 14:01:06.977863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:06.976447    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f484f85fb-2stg5" podUID="ec4a5837-9d40-488d-9753-f8b584f9bc3e" containerName="manager" containerID="cri-o://f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6" gracePeriod=30
Jul 17 14:01:07.219623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.219600    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 14:01:07.330036 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.329967    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk8bm\" (UniqueName: \"kubernetes.io/projected/ec4a5837-9d40-488d-9753-f8b584f9bc3e-kube-api-access-wk8bm\") pod \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") "
Jul 17 14:01:07.330167 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.330082    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec4a5837-9d40-488d-9753-f8b584f9bc3e-webhook-cert\") pod \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\" (UID: \"ec4a5837-9d40-488d-9753-f8b584f9bc3e\") "
Jul 17 14:01:07.331844 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.331811    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4a5837-9d40-488d-9753-f8b584f9bc3e-kube-api-access-wk8bm" (OuterVolumeSpecName: "kube-api-access-wk8bm") pod "ec4a5837-9d40-488d-9753-f8b584f9bc3e" (UID: "ec4a5837-9d40-488d-9753-f8b584f9bc3e"). InnerVolumeSpecName "kube-api-access-wk8bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:01:07.332012 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.331991    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4a5837-9d40-488d-9753-f8b584f9bc3e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ec4a5837-9d40-488d-9753-f8b584f9bc3e" (UID: "ec4a5837-9d40-488d-9753-f8b584f9bc3e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:01:07.430719 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.430692    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec4a5837-9d40-488d-9753-f8b584f9bc3e-webhook-cert\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:01:07.430719 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.430716    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk8bm\" (UniqueName: \"kubernetes.io/projected/ec4a5837-9d40-488d-9753-f8b584f9bc3e-kube-api-access-wk8bm\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:01:07.998848 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.998805    2575 generic.go:356] "Generic (PLEG): container finished" podID="ec4a5837-9d40-488d-9753-f8b584f9bc3e" containerID="f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6" exitCode=0
Jul 17 14:01:07.999268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.998855    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-2stg5"
Jul 17 14:01:07.999268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.998895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-2stg5" event={"ID":"ec4a5837-9d40-488d-9753-f8b584f9bc3e","Type":"ContainerDied","Data":"f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6"}
Jul 17 14:01:07.999268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.998962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-2stg5" event={"ID":"ec4a5837-9d40-488d-9753-f8b584f9bc3e","Type":"ContainerDied","Data":"1abd9b2d03ed37d0b831c4e2a8554207853c6d7e123c3d5c995fb5925a20a176"}
Jul 17 14:01:07.999268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:07.998988    2575 scope.go:117] "RemoveContainer" containerID="f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6"
Jul 17 14:01:08.008544 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:08.008526    2575 scope.go:117] "RemoveContainer" containerID="f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6"
Jul 17 14:01:08.008832 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:01:08.008810    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6\": container with ID starting with f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6 not found: ID does not exist" containerID="f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6"
Jul 17 14:01:08.008886 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:08.008842    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6"} err="failed to get container status \"f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6\": rpc error: code = NotFound desc = could not find container \"f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6\": container with ID starting with f97a3fed7e9c803b69474a3a3d264b02b9dcf4179903d64e1a74dd3236b9f5f6 not found: ID does not exist"
Jul 17 14:01:08.020039 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:08.020016    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f484f85fb-2stg5"]
Jul 17 14:01:08.024203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:08.024183    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f484f85fb-2stg5"]
Jul 17 14:01:08.074537 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:01:08.074511    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec4a5837-9d40-488d-9753-f8b584f9bc3e" path="/var/lib/kubelet/pods/ec4a5837-9d40-488d-9753-f8b584f9bc3e/volumes"
Jul 17 14:02:08.843385 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.843343    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f484f85fb-lpfw7"]
Jul 17 14:02:08.843967 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.843942    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4a5837-9d40-488d-9753-f8b584f9bc3e" containerName="manager"
Jul 17 14:02:08.844103 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.843970    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4a5837-9d40-488d-9753-f8b584f9bc3e" containerName="manager"
Jul 17 14:02:08.844103 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.844084    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4a5837-9d40-488d-9753-f8b584f9bc3e" containerName="manager"
Jul 17 14:02:08.847177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.847153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:08.849569 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.849544    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 14:02:08.849685 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.849590    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 14:02:08.850374 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.850357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xtg2f\""
Jul 17 14:02:08.857823 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:08.857801    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f484f85fb-lpfw7"]
Jul 17 14:02:09.027797 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.027773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb399549-8176-4211-b051-bc2e63e416f6-webhook-cert\") pod \"maas-controller-f484f85fb-lpfw7\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") " pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:09.027895 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.027800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655rk\" (UniqueName: \"kubernetes.io/projected/bb399549-8176-4211-b051-bc2e63e416f6-kube-api-access-655rk\") pod \"maas-controller-f484f85fb-lpfw7\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") " pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:09.128785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.128728    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb399549-8176-4211-b051-bc2e63e416f6-webhook-cert\") pod \"maas-controller-f484f85fb-lpfw7\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") " pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:09.128785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.128755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655rk\" (UniqueName: \"kubernetes.io/projected/bb399549-8176-4211-b051-bc2e63e416f6-kube-api-access-655rk\") pod \"maas-controller-f484f85fb-lpfw7\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") " pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:09.131090 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.131071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb399549-8176-4211-b051-bc2e63e416f6-webhook-cert\") pod \"maas-controller-f484f85fb-lpfw7\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") " pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:09.137389 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.137368    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-655rk\" (UniqueName: \"kubernetes.io/projected/bb399549-8176-4211-b051-bc2e63e416f6-kube-api-access-655rk\") pod \"maas-controller-f484f85fb-lpfw7\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") " pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:09.158137 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.158114    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:09.282115 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.282091    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f484f85fb-lpfw7"]
Jul 17 14:02:09.285079 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:02:09.285055    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb399549_8176_4211_b051_bc2e63e416f6.slice/crio-acf10324d0e80a9da13d528daa0bc06aa23e2c96aa6c54f8a9d4f969536d35d5 WatchSource:0}: Error finding container acf10324d0e80a9da13d528daa0bc06aa23e2c96aa6c54f8a9d4f969536d35d5: Status 404 returned error can't find the container with id acf10324d0e80a9da13d528daa0bc06aa23e2c96aa6c54f8a9d4f969536d35d5
Jul 17 14:02:09.286636 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.286616    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 14:02:09.303108 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:09.303084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-lpfw7" event={"ID":"bb399549-8176-4211-b051-bc2e63e416f6","Type":"ContainerStarted","Data":"acf10324d0e80a9da13d528daa0bc06aa23e2c96aa6c54f8a9d4f969536d35d5"}
Jul 17 14:02:10.308808 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:10.308772    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-lpfw7" event={"ID":"bb399549-8176-4211-b051-bc2e63e416f6","Type":"ContainerStarted","Data":"b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6"}
Jul 17 14:02:10.309243 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:10.308966    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:10.329839 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:10.329795    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f484f85fb-lpfw7" podStartSLOduration=1.825386594 podStartE2EDuration="2.329782109s" totalImagesPullingTime="504.395515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:02:10.327475271 +0000 UTC m=+1008.714822710" watchObservedRunningTime="2026-07-17 14:02:10.329782109 +0000 UTC m=+1008.717129548"
Jul 17 14:02:21.315832 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:21.315754    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:25.965465 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:25.965428    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f484f85fb-lpfw7"]
Jul 17 14:02:25.965865 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:25.965614    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f484f85fb-lpfw7" podUID="bb399549-8176-4211-b051-bc2e63e416f6" containerName="manager" containerID="cri-o://b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6" gracePeriod=30
Jul 17 14:02:26.215738 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.215684    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:26.250125 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.250095    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb399549-8176-4211-b051-bc2e63e416f6-webhook-cert\") pod \"bb399549-8176-4211-b051-bc2e63e416f6\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") "
Jul 17 14:02:26.250252 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.250188    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-655rk\" (UniqueName: \"kubernetes.io/projected/bb399549-8176-4211-b051-bc2e63e416f6-kube-api-access-655rk\") pod \"bb399549-8176-4211-b051-bc2e63e416f6\" (UID: \"bb399549-8176-4211-b051-bc2e63e416f6\") "
Jul 17 14:02:26.252130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.252101    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb399549-8176-4211-b051-bc2e63e416f6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bb399549-8176-4211-b051-bc2e63e416f6" (UID: "bb399549-8176-4211-b051-bc2e63e416f6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:02:26.252245 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.252107    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb399549-8176-4211-b051-bc2e63e416f6-kube-api-access-655rk" (OuterVolumeSpecName: "kube-api-access-655rk") pod "bb399549-8176-4211-b051-bc2e63e416f6" (UID: "bb399549-8176-4211-b051-bc2e63e416f6"). InnerVolumeSpecName "kube-api-access-655rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:02:26.350765 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.350739    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb399549-8176-4211-b051-bc2e63e416f6-webhook-cert\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:02:26.350860 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.350766    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655rk\" (UniqueName: \"kubernetes.io/projected/bb399549-8176-4211-b051-bc2e63e416f6-kube-api-access-655rk\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:02:26.381337 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.381253    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb399549-8176-4211-b051-bc2e63e416f6" containerID="b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6" exitCode=0
Jul 17 14:02:26.381337 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.381313    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-lpfw7"
Jul 17 14:02:26.381462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.381343    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-lpfw7" event={"ID":"bb399549-8176-4211-b051-bc2e63e416f6","Type":"ContainerDied","Data":"b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6"}
Jul 17 14:02:26.381462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.381397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-lpfw7" event={"ID":"bb399549-8176-4211-b051-bc2e63e416f6","Type":"ContainerDied","Data":"acf10324d0e80a9da13d528daa0bc06aa23e2c96aa6c54f8a9d4f969536d35d5"}
Jul 17 14:02:26.381462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.381419    2575 scope.go:117] "RemoveContainer" containerID="b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6"
Jul 17 14:02:26.389881 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.389865    2575 scope.go:117] "RemoveContainer" containerID="b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6"
Jul 17 14:02:26.390115 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:02:26.390097    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6\": container with ID starting with b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6 not found: ID does not exist" containerID="b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6"
Jul 17 14:02:26.390177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.390122    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6"} err="failed to get container status \"b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6\": rpc error: code = NotFound desc = could not find container \"b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6\": container with ID starting with b98bb118ebbe32ddee6de3068b7040f4d93d541b17b066584585d77300a344a6 not found: ID does not exist"
Jul 17 14:02:26.402195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.402171    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f484f85fb-lpfw7"]
Jul 17 14:02:26.408576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:26.408555    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f484f85fb-lpfw7"]
Jul 17 14:02:27.213299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.213255    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f484f85fb-rj6cw"]
Jul 17 14:02:27.213810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.213788    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb399549-8176-4211-b051-bc2e63e416f6" containerName="manager"
Jul 17 14:02:27.213896 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.213813    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb399549-8176-4211-b051-bc2e63e416f6" containerName="manager"
Jul 17 14:02:27.213948 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.213933    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb399549-8176-4211-b051-bc2e63e416f6" containerName="manager"
Jul 17 14:02:27.218126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.218107    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.221090 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.221065    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 14:02:27.221196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.221136    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xtg2f\""
Jul 17 14:02:27.221196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.221144    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 14:02:27.228854 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.228831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f484f85fb-rj6cw"]
Jul 17 14:02:27.257912 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.257891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1-webhook-cert\") pod \"maas-controller-f484f85fb-rj6cw\" (UID: \"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1\") " pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.258021 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.257932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlrv\" (UniqueName: \"kubernetes.io/projected/6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1-kube-api-access-ddlrv\") pod \"maas-controller-f484f85fb-rj6cw\" (UID: \"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1\") " pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.358525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.358502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1-webhook-cert\") pod \"maas-controller-f484f85fb-rj6cw\" (UID: \"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1\") " pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.358615 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.358549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlrv\" (UniqueName: \"kubernetes.io/projected/6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1-kube-api-access-ddlrv\") pod \"maas-controller-f484f85fb-rj6cw\" (UID: \"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1\") " pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.360789 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.360762    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1-webhook-cert\") pod \"maas-controller-f484f85fb-rj6cw\" (UID: \"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1\") " pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.366536 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.366514    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlrv\" (UniqueName: \"kubernetes.io/projected/6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1-kube-api-access-ddlrv\") pod \"maas-controller-f484f85fb-rj6cw\" (UID: \"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1\") " pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.531011 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.530976    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:27.658154 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:27.658129    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f484f85fb-rj6cw"]
Jul 17 14:02:27.659376 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:02:27.659350    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a91ab0e_eca8_4ca7_bb3c_a14cffaeccc1.slice/crio-1bf7c7095a2c3380ce0cb2d368c2f32b22f15108bbc7cf108740513d137fbde6 WatchSource:0}: Error finding container 1bf7c7095a2c3380ce0cb2d368c2f32b22f15108bbc7cf108740513d137fbde6: Status 404 returned error can't find the container with id 1bf7c7095a2c3380ce0cb2d368c2f32b22f15108bbc7cf108740513d137fbde6
Jul 17 14:02:28.075694 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:28.075659    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb399549-8176-4211-b051-bc2e63e416f6" path="/var/lib/kubelet/pods/bb399549-8176-4211-b051-bc2e63e416f6/volumes"
Jul 17 14:02:28.393525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:28.393439    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-rj6cw" event={"ID":"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1","Type":"ContainerStarted","Data":"6295e4ca9a5f497cda34128d06f6e9236d471eff2455a1d21c5c47d83e468e10"}
Jul 17 14:02:28.393525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:28.393495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f484f85fb-rj6cw" event={"ID":"6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1","Type":"ContainerStarted","Data":"1bf7c7095a2c3380ce0cb2d368c2f32b22f15108bbc7cf108740513d137fbde6"}
Jul 17 14:02:28.393982 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:28.393605    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:02:28.411229 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:28.411186    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f484f85fb-rj6cw" podStartSLOduration=0.9772909 podStartE2EDuration="1.411168978s" totalImagesPullingTime="433.878078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:02:28.40963382 +0000 UTC m=+1026.796981251" watchObservedRunningTime="2026-07-17 14:02:28.411168978 +0000 UTC m=+1026.798516417"
Jul 17 14:02:39.401205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:02:39.401174    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f484f85fb-rj6cw"
Jul 17 14:05:22.037196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:05:22.037165    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:05:22.040757 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:05:22.040731    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:05:22.041762 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:05:22.041745    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:05:22.045314 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:05:22.045295    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:05:22.047311 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:05:22.047291    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:05:22.051685 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:05:22.051669    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:10:22.075952 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:10:22.075917    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:10:22.081935 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:10:22.081910    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:10:22.082379 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:10:22.082354    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:10:22.086039 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:10:22.086017    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:10:22.089606 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:10:22.089589    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:10:22.092707 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:10:22.092690    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:14:17.440249 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.440217    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"]
Jul 17 14:14:17.443563 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.443540    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.445698 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.445675    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-988d1f69-serving-cert\""
Jul 17 14:14:17.456199 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.456174    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"]
Jul 17 14:14:17.600775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.600741    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.600942 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.600834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-openshift-service-ca\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.600942 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.600856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-kube-api-access-pbqhx\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.701425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.701341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-openshift-service-ca\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.701425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.701377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-kube-api-access-pbqhx\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.701614 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.701503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.701614 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:17.701607    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-serving-cert: secret "maas-api-e2e-ait-988d1f69-serving-cert" not found
Jul 17 14:14:17.701705 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:17.701667    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls podName:e1569215-bd75-4edb-a4bd-7a6a0a50eb69 nodeName:}" failed. No retries permitted until 2026-07-17 14:14:18.201650964 +0000 UTC m=+1736.588998382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls") pod "maas-api-e2e-ait-988d1f69-756bc7557d-2m228" (UID: "e1569215-bd75-4edb-a4bd-7a6a0a50eb69") : secret "maas-api-e2e-ait-988d1f69-serving-cert" not found
Jul 17 14:14:17.701987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.701968    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-openshift-service-ca\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:17.711797 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:17.711768    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-kube-api-access-pbqhx\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:18.205539 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:18.205509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:18.207815 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:18.207789    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls\") pod \"maas-api-e2e-ait-988d1f69-756bc7557d-2m228\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:18.354512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:18.354473    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:18.486650 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:18.486619    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"]
Jul 17 14:14:18.489217 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:14:18.489189    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1569215_bd75_4edb_a4bd_7a6a0a50eb69.slice/crio-8de4deed49a514e3fc29c89be13e9ede8f8ae36f2afda121521a318b9d27562a WatchSource:0}: Error finding container 8de4deed49a514e3fc29c89be13e9ede8f8ae36f2afda121521a318b9d27562a: Status 404 returned error can't find the container with id 8de4deed49a514e3fc29c89be13e9ede8f8ae36f2afda121521a318b9d27562a
Jul 17 14:14:18.490655 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:18.490639    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 14:14:18.695062 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:18.695027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerStarted","Data":"8de4deed49a514e3fc29c89be13e9ede8f8ae36f2afda121521a318b9d27562a"}
Jul 17 14:14:19.701681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:19.701592    2575 generic.go:356] "Generic (PLEG): container finished" podID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerID="87e7069968e5b7c9b317c77b080bd5d431a3fcc4fdd49684f5daed75b2f8be93" exitCode=1
Jul 17 14:14:19.701681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:19.701645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerDied","Data":"87e7069968e5b7c9b317c77b080bd5d431a3fcc4fdd49684f5daed75b2f8be93"}
Jul 17 14:14:19.702153 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:19.701928    2575 scope.go:117] "RemoveContainer" containerID="87e7069968e5b7c9b317c77b080bd5d431a3fcc4fdd49684f5daed75b2f8be93"
Jul 17 14:14:20.707837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:20.707805    2575 generic.go:356] "Generic (PLEG): container finished" podID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerID="568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61" exitCode=1
Jul 17 14:14:20.708213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:20.707855    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerDied","Data":"568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61"}
Jul 17 14:14:20.708213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:20.707929    2575 scope.go:117] "RemoveContainer" containerID="87e7069968e5b7c9b317c77b080bd5d431a3fcc4fdd49684f5daed75b2f8be93"
Jul 17 14:14:20.708213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:20.708130    2575 scope.go:117] "RemoveContainer" containerID="568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61"
Jul 17 14:14:20.708403 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:20.708349    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:14:21.714478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:21.714454    2575 scope.go:117] "RemoveContainer" containerID="568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61"
Jul 17 14:14:21.714841 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:21.714605    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:14:23.355177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:23.355134    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:23.355557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:23.355519    2575 scope.go:117] "RemoveContainer" containerID="568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61"
Jul 17 14:14:23.355710 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:23.355689    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:14:28.355181 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:28.355156    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:28.355524 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:28.355511    2575 scope.go:117] "RemoveContainer" containerID="568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61"
Jul 17 14:14:28.355680 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:28.355662    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:14:41.069059 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:41.069031    2575 scope.go:117] "RemoveContainer" containerID="568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61"
Jul 17 14:14:41.807577 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:41.807552    2575 generic.go:356] "Generic (PLEG): container finished" podID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerID="abd6731e3676345293a85f0189fbde174f3833860b4aab002b95e25bc6b701f8" exitCode=1
Jul 17 14:14:41.807679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:41.807626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerDied","Data":"abd6731e3676345293a85f0189fbde174f3833860b4aab002b95e25bc6b701f8"}
Jul 17 14:14:41.807718 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:41.807694    2575 scope.go:117] "RemoveContainer" containerID="568226887af31059e6a2956b88ad0bf18896f75b8c64b4d63972ccfaf34c6d61"
Jul 17 14:14:41.807999 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:41.807986    2575 scope.go:117] "RemoveContainer" containerID="abd6731e3676345293a85f0189fbde174f3833860b4aab002b95e25bc6b701f8"
Jul 17 14:14:41.808216 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:41.808194    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:14:43.354945 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:43.354910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:43.355412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:43.355220    2575 scope.go:117] "RemoveContainer" containerID="abd6731e3676345293a85f0189fbde174f3833860b4aab002b95e25bc6b701f8"
Jul 17 14:14:43.355412 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:43.355393    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:14:48.355106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:48.355064    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:14:48.355589 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:14:48.355445    2575 scope.go:117] "RemoveContainer" containerID="abd6731e3676345293a85f0189fbde174f3833860b4aab002b95e25bc6b701f8"
Jul 17 14:14:48.355631 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:14:48.355602    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:03.069419 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:03.069390    2575 scope.go:117] "RemoveContainer" containerID="abd6731e3676345293a85f0189fbde174f3833860b4aab002b95e25bc6b701f8"
Jul 17 14:15:03.918130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:03.918097    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerStarted","Data":"8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"}
Jul 17 14:15:03.918321 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:03.918308    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:15:03.937171 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:03.937118    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podStartSLOduration=44.770400953 podStartE2EDuration="46.937100365s" totalImagesPullingTime="2.166699412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:14:17 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:15:03.935228514 +0000 UTC m=+1782.322576062" watchObservedRunningTime="2026-07-17 14:15:03.937100365 +0000 UTC m=+1782.324447806"
Jul 17 14:15:04.924012 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:04.923974    2575 generic.go:356] "Generic (PLEG): container finished" podID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6" exitCode=1
Jul 17 14:15:04.924421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:04.924057    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerDied","Data":"8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"}
Jul 17 14:15:04.924421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:04.924114    2575 scope.go:117] "RemoveContainer" containerID="abd6731e3676345293a85f0189fbde174f3833860b4aab002b95e25bc6b701f8"
Jul 17 14:15:04.924421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:04.924314    2575 scope.go:117] "RemoveContainer" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"
Jul 17 14:15:04.924552 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:04.924536    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:05.933849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:05.933823    2575 scope.go:117] "RemoveContainer" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"
Jul 17 14:15:05.934325 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:05.933991    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:08.355119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:08.355091    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:15:08.355516 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:08.355485    2575 scope.go:117] "RemoveContainer" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"
Jul 17 14:15:08.355673 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:08.355654    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:21.069828 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:21.069798    2575 scope.go:117] "RemoveContainer" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"
Jul 17 14:15:21.070341 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:21.070009    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:22.107229 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:22.107203    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:15:22.110740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:22.110720    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:15:22.111997 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:22.111978    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:15:22.115462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:22.115444    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:15:22.116819 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:22.116797    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:15:22.121343 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:22.121328    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:15:36.069534 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:36.069501    2575 scope.go:117] "RemoveContainer" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"
Jul 17 14:15:36.069994 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:36.069730    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:49.069323 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:49.069293    2575 scope.go:117] "RemoveContainer" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"
Jul 17 14:15:50.139867 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:50.138484    2575 generic.go:356] "Generic (PLEG): container finished" podID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef" exitCode=1
Jul 17 14:15:50.139867 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:50.138603    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerDied","Data":"406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"}
Jul 17 14:15:50.139867 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:50.138658    2575 scope.go:117] "RemoveContainer" containerID="8116ac3225aac7d747a726745a915e327d192f04ccc690ccf9fbfd22624ea3e6"
Jul 17 14:15:50.139867 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:50.139102    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:15:50.139867 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:50.139329    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:53.918885 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:53.918845    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:15:53.919378 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:53.919247    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:15:53.919475 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:53.919453    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:15:58.354744 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:58.354704    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:15:58.355157 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:15:58.355099    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:15:58.355318 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:15:58.355269    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:16:13.069652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:16:13.069623    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:16:13.070039 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:16:13.069778    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:16:28.069205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:16:28.069175    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:16:28.069633 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:16:28.069354    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:16:39.069756 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:16:39.069727    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:16:39.070147 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:16:39.069907    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:16:50.069685 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:16:50.069607    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:16:50.070042 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:16:50.069799    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:17:02.071742 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:02.071708    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:17:02.072225 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:02.071933    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:17:13.069365 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:13.069338    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:17:14.535889 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:14.535858    2575 generic.go:356] "Generic (PLEG): container finished" podID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerID="889b4359906c9d0312930a055570ed9d5c831c8bbe81e1f6272ed5da607030cd" exitCode=1
Jul 17 14:17:14.536354 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:14.535914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerDied","Data":"889b4359906c9d0312930a055570ed9d5c831c8bbe81e1f6272ed5da607030cd"}
Jul 17 14:17:14.536354 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:14.535961    2575 scope.go:117] "RemoveContainer" containerID="406c57f2e43de2b647a6541c5d0cc1b32a1e385af37dac90b4d288c8a563d7ef"
Jul 17 14:17:14.536354 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:14.536248    2575 scope.go:117] "RemoveContainer" containerID="889b4359906c9d0312930a055570ed9d5c831c8bbe81e1f6272ed5da607030cd"
Jul 17 14:17:14.536484 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:14.536468    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:17:18.355350 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:18.355318    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:17:18.355746 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:18.355654    2575 scope.go:117] "RemoveContainer" containerID="889b4359906c9d0312930a055570ed9d5c831c8bbe81e1f6272ed5da607030cd"
Jul 17 14:17:18.355845 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:18.355825    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:17:18.918896 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:18.918858    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:17:18.919194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:18.919181    2575 scope.go:117] "RemoveContainer" containerID="889b4359906c9d0312930a055570ed9d5c831c8bbe81e1f6272ed5da607030cd"
Jul 17 14:17:18.919365 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:18.919347    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-988d1f69-756bc7557d-2m228_odh-ai-gateway-infra(e1569215-bd75-4edb-a4bd-7a6a0a50eb69)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69"
Jul 17 14:17:29.705717 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.705687    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"]
Jul 17 14:17:29.834987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.834965    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:17:29.970013 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.969988    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-kube-api-access-pbqhx\") pod \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") "
Jul 17 14:17:29.970167 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.970044    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-openshift-service-ca\") pod \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") "
Jul 17 14:17:29.970167 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.970116    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls\") pod \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\" (UID: \"e1569215-bd75-4edb-a4bd-7a6a0a50eb69\") "
Jul 17 14:17:29.970497 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.970470    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1569215-bd75-4edb-a4bd-7a6a0a50eb69" (UID: "e1569215-bd75-4edb-a4bd-7a6a0a50eb69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:17:29.972165 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.972136    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1569215-bd75-4edb-a4bd-7a6a0a50eb69" (UID: "e1569215-bd75-4edb-a4bd-7a6a0a50eb69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:17:29.972258 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:29.972172    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-kube-api-access-pbqhx" (OuterVolumeSpecName: "kube-api-access-pbqhx") pod "e1569215-bd75-4edb-a4bd-7a6a0a50eb69" (UID: "e1569215-bd75-4edb-a4bd-7a6a0a50eb69"). InnerVolumeSpecName "kube-api-access-pbqhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:17:30.071005 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.070983    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:17:30.071005 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.071005    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-kube-api-access-pbqhx\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:17:30.071137 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.071016    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1569215-bd75-4edb-a4bd-7a6a0a50eb69-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:17:30.610658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.610629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228" event={"ID":"e1569215-bd75-4edb-a4bd-7a6a0a50eb69","Type":"ContainerDied","Data":"8de4deed49a514e3fc29c89be13e9ede8f8ae36f2afda121521a318b9d27562a"}
Jul 17 14:17:30.610835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.610686    2575 scope.go:117] "RemoveContainer" containerID="889b4359906c9d0312930a055570ed9d5c831c8bbe81e1f6272ed5da607030cd"
Jul 17 14:17:30.610835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.610642    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"
Jul 17 14:17:30.628671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.628646    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"]
Jul 17 14:17:30.632461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:30.632442    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-988d1f69-756bc7557d-2m228"]
Jul 17 14:17:32.075154 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:32.075121    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" path="/var/lib/kubelet/pods/e1569215-bd75-4edb-a4bd-7a6a0a50eb69/volumes"
Jul 17 14:17:37.322270 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322239    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"]
Jul 17 14:17:37.322644 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322631    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322648    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322663    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322671    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322690    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322698    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322939 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322819    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322939 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322830    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322939 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322842    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322939 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322852    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.322939 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.322860    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:17:37.327222 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.327201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.329498 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.329472    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-398a9b19-serving-cert\""
Jul 17 14:17:37.337729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.337703    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"]
Jul 17 14:17:37.431605 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.431570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f4c34-0485-400e-8f5c-d4c243a95a5f-openshift-service-ca\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.431757 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.431638    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f4c34-0485-400e-8f5c-d4c243a95a5f-maas-api-tls\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.431757 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.431694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclrt\" (UniqueName: \"kubernetes.io/projected/459f4c34-0485-400e-8f5c-d4c243a95a5f-kube-api-access-qclrt\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.533110 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.533080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclrt\" (UniqueName: \"kubernetes.io/projected/459f4c34-0485-400e-8f5c-d4c243a95a5f-kube-api-access-qclrt\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.533259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.533134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f4c34-0485-400e-8f5c-d4c243a95a5f-openshift-service-ca\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.533259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.533168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f4c34-0485-400e-8f5c-d4c243a95a5f-maas-api-tls\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.533728 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.533710    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f4c34-0485-400e-8f5c-d4c243a95a5f-openshift-service-ca\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.535479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.535459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f4c34-0485-400e-8f5c-d4c243a95a5f-maas-api-tls\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.542856 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.542832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclrt\" (UniqueName: \"kubernetes.io/projected/459f4c34-0485-400e-8f5c-d4c243a95a5f-kube-api-access-qclrt\") pod \"maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.638438 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.638377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:37.767339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:37.767308    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"]
Jul 17 14:17:37.768186 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:17:37.768155    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459f4c34_0485_400e_8f5c_d4c243a95a5f.slice/crio-145e94b5baf13411cbdceb2b1c15a48ca35a8acae046508a0749110e9ced1456 WatchSource:0}: Error finding container 145e94b5baf13411cbdceb2b1c15a48ca35a8acae046508a0749110e9ced1456: Status 404 returned error can't find the container with id 145e94b5baf13411cbdceb2b1c15a48ca35a8acae046508a0749110e9ced1456
Jul 17 14:17:38.649682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:38.649599    2575 generic.go:356] "Generic (PLEG): container finished" podID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerID="e6b6b2715dd05226c36a9c68e5d8513d0fdcd969ba0fc8d2fc14e2bef0b9693c" exitCode=1
Jul 17 14:17:38.649682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:38.649640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" event={"ID":"459f4c34-0485-400e-8f5c-d4c243a95a5f","Type":"ContainerDied","Data":"e6b6b2715dd05226c36a9c68e5d8513d0fdcd969ba0fc8d2fc14e2bef0b9693c"}
Jul 17 14:17:38.649682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:38.649677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" event={"ID":"459f4c34-0485-400e-8f5c-d4c243a95a5f","Type":"ContainerStarted","Data":"145e94b5baf13411cbdceb2b1c15a48ca35a8acae046508a0749110e9ced1456"}
Jul 17 14:17:38.650092 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:38.649864    2575 scope.go:117] "RemoveContainer" containerID="e6b6b2715dd05226c36a9c68e5d8513d0fdcd969ba0fc8d2fc14e2bef0b9693c"
Jul 17 14:17:39.656149 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:39.656067    2575 generic.go:356] "Generic (PLEG): container finished" podID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerID="78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5" exitCode=1
Jul 17 14:17:39.656149 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:39.656103    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" event={"ID":"459f4c34-0485-400e-8f5c-d4c243a95a5f","Type":"ContainerDied","Data":"78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5"}
Jul 17 14:17:39.656149 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:39.656148    2575 scope.go:117] "RemoveContainer" containerID="e6b6b2715dd05226c36a9c68e5d8513d0fdcd969ba0fc8d2fc14e2bef0b9693c"
Jul 17 14:17:39.656653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:39.656447    2575 scope.go:117] "RemoveContainer" containerID="78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5"
Jul 17 14:17:39.656707 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:39.656685    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:17:40.662349 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:40.662324    2575 scope.go:117] "RemoveContainer" containerID="78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5"
Jul 17 14:17:40.662735 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:40.662499    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:17:42.639296 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:42.639246    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:42.639770 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:42.639740    2575 scope.go:117] "RemoveContainer" containerID="78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5"
Jul 17 14:17:42.639956 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:42.639932    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:17:47.639100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:47.639068    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:17:47.639498 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:17:47.639461    2575 scope.go:117] "RemoveContainer" containerID="78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5"
Jul 17 14:17:47.639660 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:17:47.639640    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:03.069814 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:03.069782    2575 scope.go:117] "RemoveContainer" containerID="78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5"
Jul 17 14:18:03.778224 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:03.778192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" event={"ID":"459f4c34-0485-400e-8f5c-d4c243a95a5f","Type":"ContainerStarted","Data":"23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"}
Jul 17 14:18:03.778409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:03.778394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:18:03.796681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:03.796641    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podStartSLOduration=25.695129115 podStartE2EDuration="26.796620387s" totalImagesPullingTime="1.101491272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:17:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:18:03.795818107 +0000 UTC m=+1962.183165548" watchObservedRunningTime="2026-07-17 14:18:03.796620387 +0000 UTC m=+1962.183967826"
Jul 17 14:18:04.784463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:04.784428    2575 generic.go:356] "Generic (PLEG): container finished" podID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerID="23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b" exitCode=1
Jul 17 14:18:04.784870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:04.784506    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" event={"ID":"459f4c34-0485-400e-8f5c-d4c243a95a5f","Type":"ContainerDied","Data":"23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"}
Jul 17 14:18:04.784870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:04.784559    2575 scope.go:117] "RemoveContainer" containerID="78b720f749a72b1bbac7ec89034cfa6fc636571597316984337b8c2db488f5a5"
Jul 17 14:18:04.784870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:04.784753    2575 scope.go:117] "RemoveContainer" containerID="23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"
Jul 17 14:18:04.784996 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:04.784949    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:05.791223 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:05.791193    2575 scope.go:117] "RemoveContainer" containerID="23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"
Jul 17 14:18:05.791617 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:05.791420    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:07.639553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:07.639524    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:18:07.639911 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:07.639874    2575 scope.go:117] "RemoveContainer" containerID="23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"
Jul 17 14:18:07.640048 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:07.640029    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:20.069300 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:20.069205    2575 scope.go:117] "RemoveContainer" containerID="23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"
Jul 17 14:18:20.069647 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:20.069429    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:32.071768 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:32.071737    2575 scope.go:117] "RemoveContainer" containerID="23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"
Jul 17 14:18:32.919214 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:32.919184    2575 generic.go:356] "Generic (PLEG): container finished" podID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerID="f21ecee79a128ef31f4aafb318f0da92cb612e3b5df2920f143f826b128c250f" exitCode=1
Jul 17 14:18:32.919363 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:32.919257    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" event={"ID":"459f4c34-0485-400e-8f5c-d4c243a95a5f","Type":"ContainerDied","Data":"f21ecee79a128ef31f4aafb318f0da92cb612e3b5df2920f143f826b128c250f"}
Jul 17 14:18:32.919363 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:32.919335    2575 scope.go:117] "RemoveContainer" containerID="23fc43a93e4e3fba89b86cfd6f4bc5d3eb809f052abe541eea4c6b4dbcdbbd3b"
Jul 17 14:18:32.919659 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:32.919634    2575 scope.go:117] "RemoveContainer" containerID="f21ecee79a128ef31f4aafb318f0da92cb612e3b5df2920f143f826b128c250f"
Jul 17 14:18:32.919832 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:32.919813    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:33.779060 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:33.779028    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:18:33.925433 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:33.925395    2575 scope.go:117] "RemoveContainer" containerID="f21ecee79a128ef31f4aafb318f0da92cb612e3b5df2920f143f826b128c250f"
Jul 17 14:18:33.925584 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:33.925549    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:37.638663 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:37.638633    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:18:37.639051 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:37.639000    2575 scope.go:117] "RemoveContainer" containerID="f21ecee79a128ef31f4aafb318f0da92cb612e3b5df2920f143f826b128c250f"
Jul 17 14:18:37.639187 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:37.639166    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc_odh-ai-gateway-infra(459f4c34-0485-400e-8f5c-d4c243a95a5f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f"
Jul 17 14:18:47.184342 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.184305    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"]
Jul 17 14:18:47.321473 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.321448    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:18:47.506710 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.506680    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f4c34-0485-400e-8f5c-d4c243a95a5f-maas-api-tls\") pod \"459f4c34-0485-400e-8f5c-d4c243a95a5f\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") "
Jul 17 14:18:47.506880 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.506747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f4c34-0485-400e-8f5c-d4c243a95a5f-openshift-service-ca\") pod \"459f4c34-0485-400e-8f5c-d4c243a95a5f\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") "
Jul 17 14:18:47.506880 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.506785    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qclrt\" (UniqueName: \"kubernetes.io/projected/459f4c34-0485-400e-8f5c-d4c243a95a5f-kube-api-access-qclrt\") pod \"459f4c34-0485-400e-8f5c-d4c243a95a5f\" (UID: \"459f4c34-0485-400e-8f5c-d4c243a95a5f\") "
Jul 17 14:18:47.507069 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.507042    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459f4c34-0485-400e-8f5c-d4c243a95a5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "459f4c34-0485-400e-8f5c-d4c243a95a5f" (UID: "459f4c34-0485-400e-8f5c-d4c243a95a5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:18:47.508779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.508748    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459f4c34-0485-400e-8f5c-d4c243a95a5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "459f4c34-0485-400e-8f5c-d4c243a95a5f" (UID: "459f4c34-0485-400e-8f5c-d4c243a95a5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:18:47.508878 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.508788    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459f4c34-0485-400e-8f5c-d4c243a95a5f-kube-api-access-qclrt" (OuterVolumeSpecName: "kube-api-access-qclrt") pod "459f4c34-0485-400e-8f5c-d4c243a95a5f" (UID: "459f4c34-0485-400e-8f5c-d4c243a95a5f"). InnerVolumeSpecName "kube-api-access-qclrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:18:47.608071 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.608047    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f4c34-0485-400e-8f5c-d4c243a95a5f-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:18:47.608164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.608073    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclrt\" (UniqueName: \"kubernetes.io/projected/459f4c34-0485-400e-8f5c-d4c243a95a5f-kube-api-access-qclrt\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:18:47.608164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.608087    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f4c34-0485-400e-8f5c-d4c243a95a5f-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:18:47.997080 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.997052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc" event={"ID":"459f4c34-0485-400e-8f5c-d4c243a95a5f","Type":"ContainerDied","Data":"145e94b5baf13411cbdceb2b1c15a48ca35a8acae046508a0749110e9ced1456"}
Jul 17 14:18:47.997080 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.997082    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"
Jul 17 14:18:47.997325 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:47.997110    2575 scope.go:117] "RemoveContainer" containerID="f21ecee79a128ef31f4aafb318f0da92cb612e3b5df2920f143f826b128c250f"
Jul 17 14:18:48.019103 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:48.019075    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"]
Jul 17 14:18:48.022705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:48.022683    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-398a9b19-7c66c488bb-4hwvc"]
Jul 17 14:18:48.074965 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:48.074941    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" path="/var/lib/kubelet/pods/459f4c34-0485-400e-8f5c-d4c243a95a5f/volumes"
Jul 17 14:18:52.649463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649431    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"]
Jul 17 14:18:52.649836 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649805    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:18:52.649836 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649817    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:18:52.649836 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649826    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:18:52.649836 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649831    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649839    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649845    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649854    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649860    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649871    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649879    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649892    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649897    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649903    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.649966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649908    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.650261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649974    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.650261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649985    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.650261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649991    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1569215-bd75-4edb-a4bd-7a6a0a50eb69" containerName="maas-api"
Jul 17 14:18:52.650261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.649998    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:52.654510 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.654493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"
Jul 17 14:18:52.656817 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.656801    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:18:52.661992 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.661969    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"]
Jul 17 14:18:52.850416 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.850393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb83ba77-f78c-4aa5-8748-dedb8bddb002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9\" (UID: \"bb83ba77-f78c-4aa5-8748-dedb8bddb002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"
Jul 17 14:18:52.951657 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.951595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb83ba77-f78c-4aa5-8748-dedb8bddb002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9\" (UID: \"bb83ba77-f78c-4aa5-8748-dedb8bddb002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"
Jul 17 14:18:52.952152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.952133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb83ba77-f78c-4aa5-8748-dedb8bddb002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9\" (UID: \"bb83ba77-f78c-4aa5-8748-dedb8bddb002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"
Jul 17 14:18:52.965654 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:52.965634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"
Jul 17 14:18:53.093038 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:53.093016    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"]
Jul 17 14:18:53.095095 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:18:53.095065    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb83ba77_f78c_4aa5_8748_dedb8bddb002.slice/crio-57a357df24c6b184418cf33205afe90d876c95ce1f8cc00920377b7fba1a0282 WatchSource:0}: Error finding container 57a357df24c6b184418cf33205afe90d876c95ce1f8cc00920377b7fba1a0282: Status 404 returned error can't find the container with id 57a357df24c6b184418cf33205afe90d876c95ce1f8cc00920377b7fba1a0282
Jul 17 14:18:54.028209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:54.028174    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" event={"ID":"bb83ba77-f78c-4aa5-8748-dedb8bddb002","Type":"ContainerStarted","Data":"57a357df24c6b184418cf33205afe90d876c95ce1f8cc00920377b7fba1a0282"}
Jul 17 14:18:55.842018 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.841987    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"]
Jul 17 14:18:55.842614 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.842598    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="459f4c34-0485-400e-8f5c-d4c243a95a5f" containerName="maas-api"
Jul 17 14:18:55.845768 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.845749    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.849415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.849393    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d7874222-serving-cert\""
Jul 17 14:18:55.860751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.860725    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"]
Jul 17 14:18:55.875419 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.875395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdf28e-11ac-4a4b-a502-50f89acc7093-openshift-service-ca\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.875834 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.875575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmm9\" (UniqueName: \"kubernetes.io/projected/c7cdf28e-11ac-4a4b-a502-50f89acc7093-kube-api-access-rtmm9\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.875924 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.875893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cdf28e-11ac-4a4b-a502-50f89acc7093-maas-api-tls\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.977080 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.977053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdf28e-11ac-4a4b-a502-50f89acc7093-openshift-service-ca\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.977080 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.977082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmm9\" (UniqueName: \"kubernetes.io/projected/c7cdf28e-11ac-4a4b-a502-50f89acc7093-kube-api-access-rtmm9\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.977221 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.977116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cdf28e-11ac-4a4b-a502-50f89acc7093-maas-api-tls\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.977883 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.977867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdf28e-11ac-4a4b-a502-50f89acc7093-openshift-service-ca\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.979519 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.979503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cdf28e-11ac-4a4b-a502-50f89acc7093-maas-api-tls\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:55.988060 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:55.988040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmm9\" (UniqueName: \"kubernetes.io/projected/c7cdf28e-11ac-4a4b-a502-50f89acc7093-kube-api-access-rtmm9\") pod \"maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:56.156771 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:56.156703    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:18:56.294295 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:56.294237    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"]
Jul 17 14:18:56.297105 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:18:56.297081    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cdf28e_11ac_4a4b_a502_50f89acc7093.slice/crio-6bbf44ec9ae818464a4ff2e8fe39e2897abda981a1ed03936e1f7bd6acbfc366 WatchSource:0}: Error finding container 6bbf44ec9ae818464a4ff2e8fe39e2897abda981a1ed03936e1f7bd6acbfc366: Status 404 returned error can't find the container with id 6bbf44ec9ae818464a4ff2e8fe39e2897abda981a1ed03936e1f7bd6acbfc366
Jul 17 14:18:57.044111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:57.044076    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerID="ceaa2edb28d0cf86657a6e9009276a9ca209623c3d51febaaa4058e1fdeaa97b" exitCode=1
Jul 17 14:18:57.044462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:57.044154    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"ceaa2edb28d0cf86657a6e9009276a9ca209623c3d51febaaa4058e1fdeaa97b"}
Jul 17 14:18:57.044462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:57.044204    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerStarted","Data":"6bbf44ec9ae818464a4ff2e8fe39e2897abda981a1ed03936e1f7bd6acbfc366"}
Jul 17 14:18:57.044462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:57.044365    2575 scope.go:117] "RemoveContainer" containerID="ceaa2edb28d0cf86657a6e9009276a9ca209623c3d51febaaa4058e1fdeaa97b"
Jul 17 14:18:58.051107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:58.051036    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerID="3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c" exitCode=1
Jul 17 14:18:58.051107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:58.051099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c"}
Jul 17 14:18:58.051545 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:58.051144    2575 scope.go:117] "RemoveContainer" containerID="ceaa2edb28d0cf86657a6e9009276a9ca209623c3d51febaaa4058e1fdeaa97b"
Jul 17 14:18:58.051545 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:58.051318    2575 scope.go:117] "RemoveContainer" containerID="3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c"
Jul 17 14:18:58.051545 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:58.051506    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:18:59.056672 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:18:59.056646    2575 scope.go:117] "RemoveContainer" containerID="3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c"
Jul 17 14:18:59.057010 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:18:59.056862    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:01.068262 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:01.068228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" event={"ID":"bb83ba77-f78c-4aa5-8748-dedb8bddb002","Type":"ContainerStarted","Data":"f9bb21064e57101cce59dfe8730e84790e98002bdcd7d73b20d62f2267c3ad8b"}
Jul 17 14:19:01.084470 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:01.084422    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" podStartSLOduration=1.6002888450000001 podStartE2EDuration="9.084407107s" totalImagesPullingTime="7.484118262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:19:01.082519899 +0000 UTC m=+2019.469867341" watchObservedRunningTime="2026-07-17 14:19:01.084407107 +0000 UTC m=+2019.471754547"
Jul 17 14:19:01.157085 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:01.157062    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:19:01.157426 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:01.157413    2575 scope.go:117] "RemoveContainer" containerID="3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c"
Jul 17 14:19:01.157582 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:01.157564    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:02.076258 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:02.076223    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerID="f9bb21064e57101cce59dfe8730e84790e98002bdcd7d73b20d62f2267c3ad8b" exitCode=6
Jul 17 14:19:02.076648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:02.076308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" event={"ID":"bb83ba77-f78c-4aa5-8748-dedb8bddb002","Type":"ContainerDied","Data":"f9bb21064e57101cce59dfe8730e84790e98002bdcd7d73b20d62f2267c3ad8b"}
Jul 17 14:19:02.076648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:02.076480    2575 scope.go:117] "RemoveContainer" containerID="f9bb21064e57101cce59dfe8730e84790e98002bdcd7d73b20d62f2267c3ad8b"
Jul 17 14:19:03.082413 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:03.082336    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerID="7fede858fe86c678ab99ec4bcd038c619300f2e0194d500e34926f01cc00ee48" exitCode=6
Jul 17 14:19:03.082810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:03.082424    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" event={"ID":"bb83ba77-f78c-4aa5-8748-dedb8bddb002","Type":"ContainerDied","Data":"7fede858fe86c678ab99ec4bcd038c619300f2e0194d500e34926f01cc00ee48"}
Jul 17 14:19:03.082810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:03.082492    2575 scope.go:117] "RemoveContainer" containerID="f9bb21064e57101cce59dfe8730e84790e98002bdcd7d73b20d62f2267c3ad8b"
Jul 17 14:19:03.082810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:03.082553    2575 scope.go:117] "RemoveContainer" containerID="7fede858fe86c678ab99ec4bcd038c619300f2e0194d500e34926f01cc00ee48"
Jul 17 14:19:03.082810 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:03.082723    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9_odh-ai-gateway-infra(bb83ba77-f78c-4aa5-8748-dedb8bddb002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002"
Jul 17 14:19:04.087557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:04.087530    2575 scope.go:117] "RemoveContainer" containerID="7fede858fe86c678ab99ec4bcd038c619300f2e0194d500e34926f01cc00ee48"
Jul 17 14:19:04.087928 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:04.087689    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9_odh-ai-gateway-infra(bb83ba77-f78c-4aa5-8748-dedb8bddb002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002"
Jul 17 14:19:06.157064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:06.157032    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:19:06.157532 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:06.157368    2575 scope.go:117] "RemoveContainer" containerID="3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c"
Jul 17 14:19:06.157532 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:06.157520    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:17.068944 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:17.068916    2575 scope.go:117] "RemoveContainer" containerID="7fede858fe86c678ab99ec4bcd038c619300f2e0194d500e34926f01cc00ee48"
Jul 17 14:19:18.068890 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:18.068825    2575 scope.go:117] "RemoveContainer" containerID="3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c"
Jul 17 14:19:18.149445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:18.149416    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerID="000be17a27d5d3a97015f5155039d27475675dea032e39a28b18f02f88546151" exitCode=6
Jul 17 14:19:18.149785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:18.149461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" event={"ID":"bb83ba77-f78c-4aa5-8748-dedb8bddb002","Type":"ContainerDied","Data":"000be17a27d5d3a97015f5155039d27475675dea032e39a28b18f02f88546151"}
Jul 17 14:19:18.149785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:18.149503    2575 scope.go:117] "RemoveContainer" containerID="7fede858fe86c678ab99ec4bcd038c619300f2e0194d500e34926f01cc00ee48"
Jul 17 14:19:18.149785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:18.149730    2575 scope.go:117] "RemoveContainer" containerID="000be17a27d5d3a97015f5155039d27475675dea032e39a28b18f02f88546151"
Jul 17 14:19:18.149924 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:18.149908    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9_odh-ai-gateway-infra(bb83ba77-f78c-4aa5-8748-dedb8bddb002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002"
Jul 17 14:19:19.157061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.157029    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerID="558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5" exitCode=1
Jul 17 14:19:19.157519 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.157108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5"}
Jul 17 14:19:19.157519 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.157198    2575 scope.go:117] "RemoveContainer" containerID="3d7287cbd9a72e4b054765f5699ca98a7fe2926f46e314b3569a7f9a244ea02c"
Jul 17 14:19:19.157633 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.157528    2575 scope.go:117] "RemoveContainer" containerID="558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5"
Jul 17 14:19:19.158099 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:19.157727    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:19.173165 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.173141    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"]
Jul 17 14:19:19.305355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.305331    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"
Jul 17 14:19:19.359788 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.359762    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb83ba77-f78c-4aa5-8748-dedb8bddb002-maas-api-service-ca\") pod \"bb83ba77-f78c-4aa5-8748-dedb8bddb002\" (UID: \"bb83ba77-f78c-4aa5-8748-dedb8bddb002\") "
Jul 17 14:19:19.360064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.360043    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb83ba77-f78c-4aa5-8748-dedb8bddb002-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb83ba77-f78c-4aa5-8748-dedb8bddb002" (UID: "bb83ba77-f78c-4aa5-8748-dedb8bddb002"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:19:19.460619 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:19.460543    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb83ba77-f78c-4aa5-8748-dedb8bddb002-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:19:20.164892 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:20.164862    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"
Jul 17 14:19:20.165318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:20.164886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9" event={"ID":"bb83ba77-f78c-4aa5-8748-dedb8bddb002","Type":"ContainerDied","Data":"57a357df24c6b184418cf33205afe90d876c95ce1f8cc00920377b7fba1a0282"}
Jul 17 14:19:20.165318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:20.164952    2575 scope.go:117] "RemoveContainer" containerID="000be17a27d5d3a97015f5155039d27475675dea032e39a28b18f02f88546151"
Jul 17 14:19:20.180509 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:20.180488    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"]
Jul 17 14:19:20.185732 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:20.185707    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-6ltq9"]
Jul 17 14:19:21.157295 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:21.157255    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:19:21.157602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:21.157587    2575 scope.go:117] "RemoveContainer" containerID="558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5"
Jul 17 14:19:21.157768 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:21.157750    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:22.075463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:22.075436    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" path="/var/lib/kubelet/pods/bb83ba77-f78c-4aa5-8748-dedb8bddb002/volumes"
Jul 17 14:19:26.157499 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:26.157474    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:19:26.157837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:26.157732    2575 scope.go:117] "RemoveContainer" containerID="558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5"
Jul 17 14:19:26.157891 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:26.157863    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:37.068914 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:37.068887    2575 scope.go:117] "RemoveContainer" containerID="558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5"
Jul 17 14:19:37.069333 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:37.069053    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:48.069761 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:48.069690    2575 scope.go:117] "RemoveContainer" containerID="558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5"
Jul 17 14:19:48.070228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:48.069963    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 14:19:49.300228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:49.300197    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479" exitCode=1
Jul 17 14:19:49.300635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:49.300268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"}
Jul 17 14:19:49.300635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:49.300340    2575 scope.go:117] "RemoveContainer" containerID="558b14b2ec98dd7403c723a1d165e39f37c69ba003266063bf384841e4559be5"
Jul 17 14:19:49.300739 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:49.300643    2575 scope.go:117] "RemoveContainer" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"
Jul 17 14:19:49.300876 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:49.300854    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:51.157000 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:51.156969    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:19:51.157452 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:51.157247    2575 scope.go:117] "RemoveContainer" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"
Jul 17 14:19:51.157452 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:51.157407    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:52.695983 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.695950    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"]
Jul 17 14:19:52.696381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696323    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696335    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696358    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696363    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696371    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696376    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696431    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696439    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.696600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.696446    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb83ba77-f78c-4aa5-8748-dedb8bddb002" containerName="revoke-keys"
Jul 17 14:19:52.700737 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.700719    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"
Jul 17 14:19:52.703030 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.703010    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:19:52.707326 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.707299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"]
Jul 17 14:19:52.815317 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.815273    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c95b3e-bec9-487e-bb4c-7691ed808841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-z96pf\" (UID: \"22c95b3e-bec9-487e-bb4c-7691ed808841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"
Jul 17 14:19:52.915771 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.915749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c95b3e-bec9-487e-bb4c-7691ed808841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-z96pf\" (UID: \"22c95b3e-bec9-487e-bb4c-7691ed808841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"
Jul 17 14:19:52.916371 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:52.916350    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c95b3e-bec9-487e-bb4c-7691ed808841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-z96pf\" (UID: \"22c95b3e-bec9-487e-bb4c-7691ed808841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"
Jul 17 14:19:53.011930 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:53.011911    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"
Jul 17 14:19:53.142894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:53.142728    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"]
Jul 17 14:19:53.145477 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:19:53.145447    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c95b3e_bec9_487e_bb4c_7691ed808841.slice/crio-ce0a393e42984aca888b447674de28dbc2a0086d32f3847365888c4d6833e61c WatchSource:0}: Error finding container ce0a393e42984aca888b447674de28dbc2a0086d32f3847365888c4d6833e61c: Status 404 returned error can't find the container with id ce0a393e42984aca888b447674de28dbc2a0086d32f3847365888c4d6833e61c
Jul 17 14:19:53.320485 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:53.320426    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" event={"ID":"22c95b3e-bec9-487e-bb4c-7691ed808841","Type":"ContainerStarted","Data":"ce0a393e42984aca888b447674de28dbc2a0086d32f3847365888c4d6833e61c"}
Jul 17 14:19:54.326890 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:54.326859    2575 generic.go:356] "Generic (PLEG): container finished" podID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerID="a9692e4d52a981b78298b8cda8b8e7d7c0c5ff8444e37ff33e9d6bb41ef806ab" exitCode=6
Jul 17 14:19:54.327301 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:54.326913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" event={"ID":"22c95b3e-bec9-487e-bb4c-7691ed808841","Type":"ContainerDied","Data":"a9692e4d52a981b78298b8cda8b8e7d7c0c5ff8444e37ff33e9d6bb41ef806ab"}
Jul 17 14:19:54.327301 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:54.327096    2575 scope.go:117] "RemoveContainer" containerID="a9692e4d52a981b78298b8cda8b8e7d7c0c5ff8444e37ff33e9d6bb41ef806ab"
Jul 17 14:19:55.334734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:55.334657    2575 generic.go:356] "Generic (PLEG): container finished" podID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerID="91c5bc41242edd1e3898a5366b4f67ef924dc6b7ca92224240dce6075a3d6253" exitCode=6
Jul 17 14:19:55.335092 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:55.334732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" event={"ID":"22c95b3e-bec9-487e-bb4c-7691ed808841","Type":"ContainerDied","Data":"91c5bc41242edd1e3898a5366b4f67ef924dc6b7ca92224240dce6075a3d6253"}
Jul 17 14:19:55.335092 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:55.334816    2575 scope.go:117] "RemoveContainer" containerID="a9692e4d52a981b78298b8cda8b8e7d7c0c5ff8444e37ff33e9d6bb41ef806ab"
Jul 17 14:19:55.335092 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:55.334980    2575 scope.go:117] "RemoveContainer" containerID="91c5bc41242edd1e3898a5366b4f67ef924dc6b7ca92224240dce6075a3d6253"
Jul 17 14:19:55.335225 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:55.335167    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-z96pf_odh-ai-gateway-infra(22c95b3e-bec9-487e-bb4c-7691ed808841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841"
Jul 17 14:19:56.157193 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:56.157147    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:19:56.157529 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:56.157515    2575 scope.go:117] "RemoveContainer" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"
Jul 17 14:19:56.157679 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:56.157662    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:19:56.341316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:19:56.341288    2575 scope.go:117] "RemoveContainer" containerID="91c5bc41242edd1e3898a5366b4f67ef924dc6b7ca92224240dce6075a3d6253"
Jul 17 14:19:56.341644 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:19:56.341429    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-z96pf_odh-ai-gateway-infra(22c95b3e-bec9-487e-bb4c-7691ed808841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841"
Jul 17 14:20:01.008294 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.008246    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"]
Jul 17 14:20:01.011847 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.011829    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"
Jul 17 14:20:01.019866 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.019840    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"]
Jul 17 14:20:01.178203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.178172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713920c2-c97f-438a-bb97-cb4d82601b96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-r6qkb\" (UID: \"713920c2-c97f-438a-bb97-cb4d82601b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"
Jul 17 14:20:01.278792 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.278721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713920c2-c97f-438a-bb97-cb4d82601b96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-r6qkb\" (UID: \"713920c2-c97f-438a-bb97-cb4d82601b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"
Jul 17 14:20:01.279324 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.279308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713920c2-c97f-438a-bb97-cb4d82601b96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-r6qkb\" (UID: \"713920c2-c97f-438a-bb97-cb4d82601b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"
Jul 17 14:20:01.324291 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.324260    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"
Jul 17 14:20:01.461924 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:01.461899    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"]
Jul 17 14:20:01.463619 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:20:01.463593    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713920c2_c97f_438a_bb97_cb4d82601b96.slice/crio-6acbdf5cade658c79f2c4be23469e48a934e2b5356b619faad4ad59393e9c44c WatchSource:0}: Error finding container 6acbdf5cade658c79f2c4be23469e48a934e2b5356b619faad4ad59393e9c44c: Status 404 returned error can't find the container with id 6acbdf5cade658c79f2c4be23469e48a934e2b5356b619faad4ad59393e9c44c
Jul 17 14:20:02.371229 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:02.371156    2575 generic.go:356] "Generic (PLEG): container finished" podID="713920c2-c97f-438a-bb97-cb4d82601b96" containerID="dffb4f1d3b8ecf5f5e03a20cdac4ea8910d71774da9bc7a2390e191894db1f13" exitCode=7
Jul 17 14:20:02.371593 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:02.371238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" event={"ID":"713920c2-c97f-438a-bb97-cb4d82601b96","Type":"ContainerDied","Data":"dffb4f1d3b8ecf5f5e03a20cdac4ea8910d71774da9bc7a2390e191894db1f13"}
Jul 17 14:20:02.371593 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:02.371301    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" event={"ID":"713920c2-c97f-438a-bb97-cb4d82601b96","Type":"ContainerStarted","Data":"6acbdf5cade658c79f2c4be23469e48a934e2b5356b619faad4ad59393e9c44c"}
Jul 17 14:20:02.371593 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:02.371401    2575 scope.go:117] "RemoveContainer" containerID="dffb4f1d3b8ecf5f5e03a20cdac4ea8910d71774da9bc7a2390e191894db1f13"
Jul 17 14:20:03.377380 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:03.377350    2575 generic.go:356] "Generic (PLEG): container finished" podID="713920c2-c97f-438a-bb97-cb4d82601b96" containerID="d078e67ca9e4e60c8fdea66dc0910a820f7e225e61b544a36331b719a8afb096" exitCode=7
Jul 17 14:20:03.377816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:03.377403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" event={"ID":"713920c2-c97f-438a-bb97-cb4d82601b96","Type":"ContainerDied","Data":"d078e67ca9e4e60c8fdea66dc0910a820f7e225e61b544a36331b719a8afb096"}
Jul 17 14:20:03.377816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:03.377448    2575 scope.go:117] "RemoveContainer" containerID="dffb4f1d3b8ecf5f5e03a20cdac4ea8910d71774da9bc7a2390e191894db1f13"
Jul 17 14:20:03.377816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:03.377651    2575 scope.go:117] "RemoveContainer" containerID="d078e67ca9e4e60c8fdea66dc0910a820f7e225e61b544a36331b719a8afb096"
Jul 17 14:20:03.377955 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:03.377860    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-r6qkb_odh-ai-gateway-infra(713920c2-c97f-438a-bb97-cb4d82601b96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" podUID="713920c2-c97f-438a-bb97-cb4d82601b96"
Jul 17 14:20:04.383568 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:04.383544    2575 scope.go:117] "RemoveContainer" containerID="d078e67ca9e4e60c8fdea66dc0910a820f7e225e61b544a36331b719a8afb096"
Jul 17 14:20:04.383926 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:04.383682    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-r6qkb_odh-ai-gateway-infra(713920c2-c97f-438a-bb97-cb4d82601b96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" podUID="713920c2-c97f-438a-bb97-cb4d82601b96"
Jul 17 14:20:09.069489 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:09.069461    2575 scope.go:117] "RemoveContainer" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"
Jul 17 14:20:09.069951 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:09.069666    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:20:10.069654 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:10.069623    2575 scope.go:117] "RemoveContainer" containerID="91c5bc41242edd1e3898a5366b4f67ef924dc6b7ca92224240dce6075a3d6253"
Jul 17 14:20:10.414329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:10.414247    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" event={"ID":"22c95b3e-bec9-487e-bb4c-7691ed808841","Type":"ContainerStarted","Data":"5bc16f7bb1675359170c91b5a294dfd9c0b63551ceff17270afaa07d2276eff7"}
Jul 17 14:20:10.432219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:10.432146    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" podStartSLOduration=18.43213267 podStartE2EDuration="18.43213267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:20:10.430773924 +0000 UTC m=+2088.818121363" watchObservedRunningTime="2026-07-17 14:20:10.43213267 +0000 UTC m=+2088.819480423"
Jul 17 14:20:11.420856 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:11.420826    2575 generic.go:356] "Generic (PLEG): container finished" podID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerID="5bc16f7bb1675359170c91b5a294dfd9c0b63551ceff17270afaa07d2276eff7" exitCode=6
Jul 17 14:20:11.421304 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:11.420895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" event={"ID":"22c95b3e-bec9-487e-bb4c-7691ed808841","Type":"ContainerDied","Data":"5bc16f7bb1675359170c91b5a294dfd9c0b63551ceff17270afaa07d2276eff7"}
Jul 17 14:20:11.421304 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:11.420952    2575 scope.go:117] "RemoveContainer" containerID="91c5bc41242edd1e3898a5366b4f67ef924dc6b7ca92224240dce6075a3d6253"
Jul 17 14:20:11.421304 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:11.421206    2575 scope.go:117] "RemoveContainer" containerID="5bc16f7bb1675359170c91b5a294dfd9c0b63551ceff17270afaa07d2276eff7"
Jul 17 14:20:11.421476 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:11.421423    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-z96pf_odh-ai-gateway-infra(22c95b3e-bec9-487e-bb4c-7691ed808841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841"
Jul 17 14:20:11.440577 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:11.440555    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"]
Jul 17 14:20:12.562932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:12.562904    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"
Jul 17 14:20:12.575471 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:12.575445    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c95b3e-bec9-487e-bb4c-7691ed808841-maas-api-service-ca\") pod \"22c95b3e-bec9-487e-bb4c-7691ed808841\" (UID: \"22c95b3e-bec9-487e-bb4c-7691ed808841\") "
Jul 17 14:20:12.575740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:12.575720    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c95b3e-bec9-487e-bb4c-7691ed808841-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22c95b3e-bec9-487e-bb4c-7691ed808841" (UID: "22c95b3e-bec9-487e-bb4c-7691ed808841"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:20:12.676075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:12.676050    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c95b3e-bec9-487e-bb4c-7691ed808841-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:20:13.432036 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:13.432007    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"
Jul 17 14:20:13.432228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:13.432052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf" event={"ID":"22c95b3e-bec9-487e-bb4c-7691ed808841","Type":"ContainerDied","Data":"ce0a393e42984aca888b447674de28dbc2a0086d32f3847365888c4d6833e61c"}
Jul 17 14:20:13.432228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:13.432110    2575 scope.go:117] "RemoveContainer" containerID="5bc16f7bb1675359170c91b5a294dfd9c0b63551ceff17270afaa07d2276eff7"
Jul 17 14:20:13.455669 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:13.455649    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"]
Jul 17 14:20:13.460918 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:13.460899    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-z96pf"]
Jul 17 14:20:14.074787 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:14.074761    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" path="/var/lib/kubelet/pods/22c95b3e-bec9-487e-bb4c-7691ed808841/volumes"
Jul 17 14:20:18.069603 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:18.069571    2575 scope.go:117] "RemoveContainer" containerID="d078e67ca9e4e60c8fdea66dc0910a820f7e225e61b544a36331b719a8afb096"
Jul 17 14:20:18.456376 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:18.456346    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" event={"ID":"713920c2-c97f-438a-bb97-cb4d82601b96","Type":"ContainerStarted","Data":"e7b53295faca340fe565b9a6d8ccb08482f4e38486176daf750c8f7c6254b361"}
Jul 17 14:20:18.474820 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:18.474775    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" podStartSLOduration=18.47475867 podStartE2EDuration="18.47475867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:20:18.472517199 +0000 UTC m=+2096.859864649" watchObservedRunningTime="2026-07-17 14:20:18.47475867 +0000 UTC m=+2096.862106110"
Jul 17 14:20:19.462600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:19.462565    2575 generic.go:356] "Generic (PLEG): container finished" podID="713920c2-c97f-438a-bb97-cb4d82601b96" containerID="e7b53295faca340fe565b9a6d8ccb08482f4e38486176daf750c8f7c6254b361" exitCode=7
Jul 17 14:20:19.463014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:19.462641    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" event={"ID":"713920c2-c97f-438a-bb97-cb4d82601b96","Type":"ContainerDied","Data":"e7b53295faca340fe565b9a6d8ccb08482f4e38486176daf750c8f7c6254b361"}
Jul 17 14:20:19.463014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:19.462714    2575 scope.go:117] "RemoveContainer" containerID="d078e67ca9e4e60c8fdea66dc0910a820f7e225e61b544a36331b719a8afb096"
Jul 17 14:20:19.463014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:19.462986    2575 scope.go:117] "RemoveContainer" containerID="e7b53295faca340fe565b9a6d8ccb08482f4e38486176daf750c8f7c6254b361"
Jul 17 14:20:19.463179 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:19.463161    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-r6qkb_odh-ai-gateway-infra(713920c2-c97f-438a-bb97-cb4d82601b96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" podUID="713920c2-c97f-438a-bb97-cb4d82601b96"
Jul 17 14:20:19.488534 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:19.488509    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"]
Jul 17 14:20:20.598078 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:20.598051    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"
Jul 17 14:20:20.634530 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:20.634508    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713920c2-c97f-438a-bb97-cb4d82601b96-maas-api-service-ca\") pod \"713920c2-c97f-438a-bb97-cb4d82601b96\" (UID: \"713920c2-c97f-438a-bb97-cb4d82601b96\") "
Jul 17 14:20:20.634821 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:20.634801    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713920c2-c97f-438a-bb97-cb4d82601b96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "713920c2-c97f-438a-bb97-cb4d82601b96" (UID: "713920c2-c97f-438a-bb97-cb4d82601b96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:20:20.735706 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:20.735644    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713920c2-c97f-438a-bb97-cb4d82601b96-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:20:21.474923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:21.474897    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"
Jul 17 14:20:21.475077 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:21.474924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb" event={"ID":"713920c2-c97f-438a-bb97-cb4d82601b96","Type":"ContainerDied","Data":"6acbdf5cade658c79f2c4be23469e48a934e2b5356b619faad4ad59393e9c44c"}
Jul 17 14:20:21.475077 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:21.474984    2575 scope.go:117] "RemoveContainer" containerID="e7b53295faca340fe565b9a6d8ccb08482f4e38486176daf750c8f7c6254b361"
Jul 17 14:20:21.497529 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:21.497506    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"]
Jul 17 14:20:21.503705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:21.503684    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-r6qkb"]
Jul 17 14:20:22.075106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:22.075070    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" path="/var/lib/kubelet/pods/713920c2-c97f-438a-bb97-cb4d82601b96/volumes"
Jul 17 14:20:22.135838 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:22.135812    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:20:22.139812 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:22.139793    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:20:22.141688 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:22.141669    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:20:22.145414 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:22.145397    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:20:22.146262 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:22.146245    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:20:22.151682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:22.151665    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:20:24.069412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:24.069382    2575 scope.go:117] "RemoveContainer" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"
Jul 17 14:20:24.069868 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:24.069566    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:20:37.069257 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:37.069223    2575 scope.go:117] "RemoveContainer" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"
Jul 17 14:20:37.549635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:37.549613    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerStarted","Data":"f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"}
Jul 17 14:20:37.549814 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:37.549799    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:20:37.569869 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:37.569825    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podStartSLOduration=100.72894614 podStartE2EDuration="1m42.569810455s" totalImagesPullingTime="1.840864315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:18:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:20:37.568747987 +0000 UTC m=+2115.956095428" watchObservedRunningTime="2026-07-17 14:20:37.569810455 +0000 UTC m=+2115.957157894"
Jul 17 14:20:38.555797 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:38.555765    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6" exitCode=1
Jul 17 14:20:38.556207 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:38.555840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"}
Jul 17 14:20:38.556207 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:38.555890    2575 scope.go:117] "RemoveContainer" containerID="cdfe3566f780b663136dc68a1c2625a9d396106e484b6af63f6691434157e479"
Jul 17 14:20:38.556207 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:38.556141    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:20:38.556392 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:38.556373    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:20:39.562515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:39.562486    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:20:39.562899 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:39.562643    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:20:46.157459 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:46.157428    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:20:46.157903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:46.157756    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:20:46.157951 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:46.157898    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:20:57.069493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:20:57.069457    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:20:57.069875 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:20:57.069620    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:21:08.069674 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:08.069634    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:21:08.070025 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:21:08.069860    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:21:23.069396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:23.069324    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:21:23.069835 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:21:23.069486    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:21:35.069664 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:35.069636    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:21:35.070028 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:21:35.069810    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:21:48.069032 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:48.069001    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:21:48.069414 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:21:48.069188    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:21:51.121916 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.121878    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"]
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122309    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122321    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122335    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122340    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122349    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122354    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122361    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122369    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122378    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122383    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122465    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122475    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:21:51.122899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122483    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:21:51.122899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122489    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:21:51.122899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.122495    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:21:51.125511 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.125492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"
Jul 17 14:21:51.128211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.128190    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:21:51.138347 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.138326    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"]
Jul 17 14:21:51.238234 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.238202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5ba515-b87e-4beb-b5c6-a6eaa32ba546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-zflvp\" (UID: \"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"
Jul 17 14:21:51.339477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.339447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5ba515-b87e-4beb-b5c6-a6eaa32ba546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-zflvp\" (UID: \"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"
Jul 17 14:21:51.340001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.339983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5ba515-b87e-4beb-b5c6-a6eaa32ba546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-zflvp\" (UID: \"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"
Jul 17 14:21:51.435359 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.435299    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"
Jul 17 14:21:51.566514 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.566492    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"]
Jul 17 14:21:51.568790 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:21:51.568764    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5ba515_b87e_4beb_b5c6_a6eaa32ba546.slice/crio-4e644f67cdff9f6f081ad2075c966d9cff1b6cd6e2725e5c5b6aa4dd660f818b WatchSource:0}: Error finding container 4e644f67cdff9f6f081ad2075c966d9cff1b6cd6e2725e5c5b6aa4dd660f818b: Status 404 returned error can't find the container with id 4e644f67cdff9f6f081ad2075c966d9cff1b6cd6e2725e5c5b6aa4dd660f818b
Jul 17 14:21:51.892319 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.892268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" event={"ID":"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546","Type":"ContainerStarted","Data":"e8d296ff18d9678b53e64859afabf7ba3354011b4bd4948fb7c958212d28093e"}
Jul 17 14:21:51.892429 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.892336    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" event={"ID":"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546","Type":"ContainerStarted","Data":"4e644f67cdff9f6f081ad2075c966d9cff1b6cd6e2725e5c5b6aa4dd660f818b"}
Jul 17 14:21:51.913987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:51.913943    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" podStartSLOduration=0.913927602 podStartE2EDuration="913.927602ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:21:51.913335587 +0000 UTC m=+2190.300683026" watchObservedRunningTime="2026-07-17 14:21:51.913927602 +0000 UTC m=+2190.301275043"
Jul 17 14:21:52.898329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:52.898298    2575 generic.go:356] "Generic (PLEG): container finished" podID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerID="e8d296ff18d9678b53e64859afabf7ba3354011b4bd4948fb7c958212d28093e" exitCode=7
Jul 17 14:21:52.898329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:52.898333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" event={"ID":"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546","Type":"ContainerDied","Data":"e8d296ff18d9678b53e64859afabf7ba3354011b4bd4948fb7c958212d28093e"}
Jul 17 14:21:52.898736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:52.898529    2575 scope.go:117] "RemoveContainer" containerID="e8d296ff18d9678b53e64859afabf7ba3354011b4bd4948fb7c958212d28093e"
Jul 17 14:21:53.905076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:53.904999    2575 generic.go:356] "Generic (PLEG): container finished" podID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerID="d568aafc9070c46ca0c273bc162e1e7e3a183f68240d51e166edbfb798ef8285" exitCode=7
Jul 17 14:21:53.905446 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:53.905094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" event={"ID":"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546","Type":"ContainerDied","Data":"d568aafc9070c46ca0c273bc162e1e7e3a183f68240d51e166edbfb798ef8285"}
Jul 17 14:21:53.905446 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:53.905154    2575 scope.go:117] "RemoveContainer" containerID="e8d296ff18d9678b53e64859afabf7ba3354011b4bd4948fb7c958212d28093e"
Jul 17 14:21:53.905446 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:53.905264    2575 scope.go:117] "RemoveContainer" containerID="d568aafc9070c46ca0c273bc162e1e7e3a183f68240d51e166edbfb798ef8285"
Jul 17 14:21:53.905570 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:21:53.905513    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-zflvp_odh-ai-gateway-infra(ff5ba515-b87e-4beb-b5c6-a6eaa32ba546)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546"
Jul 17 14:21:54.917110 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:21:54.917081    2575 scope.go:117] "RemoveContainer" containerID="d568aafc9070c46ca0c273bc162e1e7e3a183f68240d51e166edbfb798ef8285"
Jul 17 14:21:54.917671 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:21:54.917360    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-zflvp_odh-ai-gateway-infra(ff5ba515-b87e-4beb-b5c6-a6eaa32ba546)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546"
Jul 17 14:22:00.069113 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:00.069088    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:22:00.945877 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:00.945845    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057" exitCode=1
Jul 17 14:22:00.946025 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:00.945919    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"}
Jul 17 14:22:00.946025 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:00.945989    2575 scope.go:117] "RemoveContainer" containerID="f5168b2b094e75ba95a90795cbefe862afe3a6f09238ebacc8134fdd55c6a6f6"
Jul 17 14:22:00.946364 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:00.946347    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:22:00.946564 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:00.946541    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:22:02.550875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:02.550844    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:22:02.551267 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:02.551192    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:22:02.551414 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:02.551393    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:22:06.156914 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:06.156885    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:22:06.157297 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:06.157167    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:22:06.157345 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:06.157331    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:22:07.069197 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:07.069168    2575 scope.go:117] "RemoveContainer" containerID="d568aafc9070c46ca0c273bc162e1e7e3a183f68240d51e166edbfb798ef8285"
Jul 17 14:22:07.989145 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:07.989114    2575 generic.go:356] "Generic (PLEG): container finished" podID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerID="4f160c7d02f87a41e5c8473743909a843a000b411dc2c7c61fb3a5aab6a5e520" exitCode=7
Jul 17 14:22:07.989546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:07.989187    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" event={"ID":"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546","Type":"ContainerDied","Data":"4f160c7d02f87a41e5c8473743909a843a000b411dc2c7c61fb3a5aab6a5e520"}
Jul 17 14:22:07.989546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:07.989247    2575 scope.go:117] "RemoveContainer" containerID="d568aafc9070c46ca0c273bc162e1e7e3a183f68240d51e166edbfb798ef8285"
Jul 17 14:22:07.989546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:07.989536    2575 scope.go:117] "RemoveContainer" containerID="4f160c7d02f87a41e5c8473743909a843a000b411dc2c7c61fb3a5aab6a5e520"
Jul 17 14:22:07.989810 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:07.989790    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-zflvp_odh-ai-gateway-infra(ff5ba515-b87e-4beb-b5c6-a6eaa32ba546)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546"
Jul 17 14:22:09.021138 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:09.021109    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"]
Jul 17 14:22:09.154461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:09.154439    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"
Jul 17 14:22:09.282152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:09.282071    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5ba515-b87e-4beb-b5c6-a6eaa32ba546-maas-api-service-ca\") pod \"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546\" (UID: \"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546\") "
Jul 17 14:22:09.282476 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:09.282456    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5ba515-b87e-4beb-b5c6-a6eaa32ba546-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" (UID: "ff5ba515-b87e-4beb-b5c6-a6eaa32ba546"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:22:09.383388 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:09.383350    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5ba515-b87e-4beb-b5c6-a6eaa32ba546-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:22:10.001005 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:10.000974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp" event={"ID":"ff5ba515-b87e-4beb-b5c6-a6eaa32ba546","Type":"ContainerDied","Data":"4e644f67cdff9f6f081ad2075c966d9cff1b6cd6e2725e5c5b6aa4dd660f818b"}
Jul 17 14:22:10.001208 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:10.001032    2575 scope.go:117] "RemoveContainer" containerID="4f160c7d02f87a41e5c8473743909a843a000b411dc2c7c61fb3a5aab6a5e520"
Jul 17 14:22:10.001208 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:10.000983    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"
Jul 17 14:22:10.023262 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:10.023235    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"]
Jul 17 14:22:10.026890 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:10.026862    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-zflvp"]
Jul 17 14:22:10.074175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:10.074146    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" path="/var/lib/kubelet/pods/ff5ba515-b87e-4beb-b5c6-a6eaa32ba546/volumes"
Jul 17 14:22:18.069740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:18.069715    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:22:18.070127 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:18.069878    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:22:32.075890 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:32.075861    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:22:32.076449 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:32.076076    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:22:41.228909 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.228875    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"]
Jul 17 14:22:41.229319 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229267    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229319 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229291    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229319 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229309    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229319 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229316    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229324    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229330    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c95b3e-bec9-487e-bb4c-7691ed808841" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229338    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229343    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229404    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="713920c2-c97f-438a-bb97-cb4d82601b96" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229415    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229422    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.229478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.229428    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5ba515-b87e-4beb-b5c6-a6eaa32ba546" containerName="revoke-keys"
Jul 17 14:22:41.232544 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.232529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"
Jul 17 14:22:41.240995 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.240976    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:22:41.286218 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.286197    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"]
Jul 17 14:22:41.351553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.351515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdefe6f5-3cf9-4776-ac3e-1c9a3c244212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-g5dgd\" (UID: \"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"
Jul 17 14:22:41.452107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.452078    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdefe6f5-3cf9-4776-ac3e-1c9a3c244212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-g5dgd\" (UID: \"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"
Jul 17 14:22:41.452667 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.452650    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdefe6f5-3cf9-4776-ac3e-1c9a3c244212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-g5dgd\" (UID: \"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"
Jul 17 14:22:41.542375 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.542319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"
Jul 17 14:22:41.683248 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:41.683209    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"]
Jul 17 14:22:41.686384 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:22:41.686358    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdefe6f5_3cf9_4776_ac3e_1c9a3c244212.slice/crio-8a955a6fd7e738705429b2af85928e9bf48e79a4715d7ff635021b1db22e0e4b WatchSource:0}: Error finding container 8a955a6fd7e738705429b2af85928e9bf48e79a4715d7ff635021b1db22e0e4b: Status 404 returned error can't find the container with id 8a955a6fd7e738705429b2af85928e9bf48e79a4715d7ff635021b1db22e0e4b
Jul 17 14:22:42.158120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:42.158026    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" event={"ID":"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212","Type":"ContainerStarted","Data":"e0d7fce98d39ad902ee985b95e1b87560cfaa687c76e332c2fc233ea92bce65f"}
Jul 17 14:22:42.158120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:42.158086    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" event={"ID":"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212","Type":"ContainerStarted","Data":"8a955a6fd7e738705429b2af85928e9bf48e79a4715d7ff635021b1db22e0e4b"}
Jul 17 14:22:42.186342 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:42.186294    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" podStartSLOduration=1.186263657 podStartE2EDuration="1.186263657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:22:42.185410696 +0000 UTC m=+2240.572758137" watchObservedRunningTime="2026-07-17 14:22:42.186263657 +0000 UTC m=+2240.573611096"
Jul 17 14:22:43.164073 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:43.164039    2575 generic.go:356] "Generic (PLEG): container finished" podID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerID="e0d7fce98d39ad902ee985b95e1b87560cfaa687c76e332c2fc233ea92bce65f" exitCode=7
Jul 17 14:22:43.164461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:43.164121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" event={"ID":"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212","Type":"ContainerDied","Data":"e0d7fce98d39ad902ee985b95e1b87560cfaa687c76e332c2fc233ea92bce65f"}
Jul 17 14:22:43.164461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:43.164309    2575 scope.go:117] "RemoveContainer" containerID="e0d7fce98d39ad902ee985b95e1b87560cfaa687c76e332c2fc233ea92bce65f"
Jul 17 14:22:44.169756 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:44.169676    2575 generic.go:356] "Generic (PLEG): container finished" podID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerID="5fd1378f614bf1ab23ce32bffe0d8e7d2f1193fdb46b458dcc2f72ffe94fb863" exitCode=7
Jul 17 14:22:44.170116 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:44.169760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" event={"ID":"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212","Type":"ContainerDied","Data":"5fd1378f614bf1ab23ce32bffe0d8e7d2f1193fdb46b458dcc2f72ffe94fb863"}
Jul 17 14:22:44.170116 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:44.169825    2575 scope.go:117] "RemoveContainer" containerID="e0d7fce98d39ad902ee985b95e1b87560cfaa687c76e332c2fc233ea92bce65f"
Jul 17 14:22:44.170116 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:44.170022    2575 scope.go:117] "RemoveContainer" containerID="5fd1378f614bf1ab23ce32bffe0d8e7d2f1193fdb46b458dcc2f72ffe94fb863"
Jul 17 14:22:44.170239 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:44.170212    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-g5dgd_odh-ai-gateway-infra(fdefe6f5-3cf9-4776-ac3e-1c9a3c244212)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212"
Jul 17 14:22:45.068909 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:45.068879    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:22:45.069077 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:45.069058    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:22:45.176156 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:45.176127    2575 scope.go:117] "RemoveContainer" containerID="5fd1378f614bf1ab23ce32bffe0d8e7d2f1193fdb46b458dcc2f72ffe94fb863"
Jul 17 14:22:45.176534 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:45.176367    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-g5dgd_odh-ai-gateway-infra(fdefe6f5-3cf9-4776-ac3e-1c9a3c244212)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212"
Jul 17 14:22:52.837887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:52.837846    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"]
Jul 17 14:22:52.842835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:52.842812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"
Jul 17 14:22:52.848971 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:52.848937    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"]
Jul 17 14:22:52.941803 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:52.941776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00374493-02dd-4e29-9348-150fbb2c5f67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt\" (UID: \"00374493-02dd-4e29-9348-150fbb2c5f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"
Jul 17 14:22:53.043002 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:53.042978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00374493-02dd-4e29-9348-150fbb2c5f67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt\" (UID: \"00374493-02dd-4e29-9348-150fbb2c5f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"
Jul 17 14:22:53.043584 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:53.043566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00374493-02dd-4e29-9348-150fbb2c5f67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt\" (UID: \"00374493-02dd-4e29-9348-150fbb2c5f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"
Jul 17 14:22:53.153431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:53.153369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"
Jul 17 14:22:53.280503 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:53.280478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"]
Jul 17 14:22:53.283008 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:22:53.282965    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00374493_02dd_4e29_9348_150fbb2c5f67.slice/crio-05807ef1c5360eb377790ebe8c2a2d571263defb0523ce6e18184edefe59ce2a WatchSource:0}: Error finding container 05807ef1c5360eb377790ebe8c2a2d571263defb0523ce6e18184edefe59ce2a: Status 404 returned error can't find the container with id 05807ef1c5360eb377790ebe8c2a2d571263defb0523ce6e18184edefe59ce2a
Jul 17 14:22:54.219166 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:54.219135    2575 generic.go:356] "Generic (PLEG): container finished" podID="00374493-02dd-4e29-9348-150fbb2c5f67" containerID="76503e02da4703440c87c45a6f2fa542025a0e8156b8283d092cceb225d3c129" exitCode=6
Jul 17 14:22:54.219630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:54.219218    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" event={"ID":"00374493-02dd-4e29-9348-150fbb2c5f67","Type":"ContainerDied","Data":"76503e02da4703440c87c45a6f2fa542025a0e8156b8283d092cceb225d3c129"}
Jul 17 14:22:54.219630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:54.219267    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" event={"ID":"00374493-02dd-4e29-9348-150fbb2c5f67","Type":"ContainerStarted","Data":"05807ef1c5360eb377790ebe8c2a2d571263defb0523ce6e18184edefe59ce2a"}
Jul 17 14:22:54.219630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:54.219442    2575 scope.go:117] "RemoveContainer" containerID="76503e02da4703440c87c45a6f2fa542025a0e8156b8283d092cceb225d3c129"
Jul 17 14:22:55.225614 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:55.225586    2575 generic.go:356] "Generic (PLEG): container finished" podID="00374493-02dd-4e29-9348-150fbb2c5f67" containerID="1c783d22b5eebc6d6b7f4e3d2e0fc8003106cf0690d215a45bd0d6a20284c14c" exitCode=6
Jul 17 14:22:55.225993 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:55.225671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" event={"ID":"00374493-02dd-4e29-9348-150fbb2c5f67","Type":"ContainerDied","Data":"1c783d22b5eebc6d6b7f4e3d2e0fc8003106cf0690d215a45bd0d6a20284c14c"}
Jul 17 14:22:55.225993 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:55.225724    2575 scope.go:117] "RemoveContainer" containerID="76503e02da4703440c87c45a6f2fa542025a0e8156b8283d092cceb225d3c129"
Jul 17 14:22:55.225993 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:55.225824    2575 scope.go:117] "RemoveContainer" containerID="1c783d22b5eebc6d6b7f4e3d2e0fc8003106cf0690d215a45bd0d6a20284c14c"
Jul 17 14:22:55.226128 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:55.226002    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt_odh-ai-gateway-infra(00374493-02dd-4e29-9348-150fbb2c5f67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" podUID="00374493-02dd-4e29-9348-150fbb2c5f67"
Jul 17 14:22:56.231569 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:22:56.231540    2575 scope.go:117] "RemoveContainer" containerID="1c783d22b5eebc6d6b7f4e3d2e0fc8003106cf0690d215a45bd0d6a20284c14c"
Jul 17 14:22:56.231944 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:22:56.231707    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt_odh-ai-gateway-infra(00374493-02dd-4e29-9348-150fbb2c5f67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" podUID="00374493-02dd-4e29-9348-150fbb2c5f67"
Jul 17 14:23:00.069740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:00.069711    2575 scope.go:117] "RemoveContainer" containerID="5fd1378f614bf1ab23ce32bffe0d8e7d2f1193fdb46b458dcc2f72ffe94fb863"
Jul 17 14:23:00.070105 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:00.069829    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:23:00.070105 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:00.069992    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:23:01.257850 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:01.257818    2575 generic.go:356] "Generic (PLEG): container finished" podID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerID="4f1e03e894e7db054698e392cdfc982661b13d92388d8d97e39bd98a4c48393d" exitCode=7
Jul 17 14:23:01.258292 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:01.257899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" event={"ID":"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212","Type":"ContainerDied","Data":"4f1e03e894e7db054698e392cdfc982661b13d92388d8d97e39bd98a4c48393d"}
Jul 17 14:23:01.258292 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:01.257979    2575 scope.go:117] "RemoveContainer" containerID="5fd1378f614bf1ab23ce32bffe0d8e7d2f1193fdb46b458dcc2f72ffe94fb863"
Jul 17 14:23:01.258292 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:01.258267    2575 scope.go:117] "RemoveContainer" containerID="4f1e03e894e7db054698e392cdfc982661b13d92388d8d97e39bd98a4c48393d"
Jul 17 14:23:01.258539 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:01.258519    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-g5dgd_odh-ai-gateway-infra(fdefe6f5-3cf9-4776-ac3e-1c9a3c244212)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212"
Jul 17 14:23:02.291053 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:02.291021    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"]
Jul 17 14:23:02.424288 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:02.424255    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"
Jul 17 14:23:02.521302 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:02.521254    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdefe6f5-3cf9-4776-ac3e-1c9a3c244212-maas-api-service-ca\") pod \"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212\" (UID: \"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212\") "
Jul 17 14:23:02.521615 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:02.521586    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdefe6f5-3cf9-4776-ac3e-1c9a3c244212-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" (UID: "fdefe6f5-3cf9-4776-ac3e-1c9a3c244212"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:23:02.622044 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:02.621970    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdefe6f5-3cf9-4776-ac3e-1c9a3c244212-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:23:03.269926 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:03.269902    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"
Jul 17 14:23:03.270111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:03.269947    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd" event={"ID":"fdefe6f5-3cf9-4776-ac3e-1c9a3c244212","Type":"ContainerDied","Data":"8a955a6fd7e738705429b2af85928e9bf48e79a4715d7ff635021b1db22e0e4b"}
Jul 17 14:23:03.270111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:03.270005    2575 scope.go:117] "RemoveContainer" containerID="4f1e03e894e7db054698e392cdfc982661b13d92388d8d97e39bd98a4c48393d"
Jul 17 14:23:03.293536 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:03.293501    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"]
Jul 17 14:23:03.296653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:03.296629    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g5dgd"]
Jul 17 14:23:04.074786 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:04.074747    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" path="/var/lib/kubelet/pods/fdefe6f5-3cf9-4776-ac3e-1c9a3c244212/volumes"
Jul 17 14:23:08.069321 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:08.069292    2575 scope.go:117] "RemoveContainer" containerID="1c783d22b5eebc6d6b7f4e3d2e0fc8003106cf0690d215a45bd0d6a20284c14c"
Jul 17 14:23:08.297943 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:08.297914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" event={"ID":"00374493-02dd-4e29-9348-150fbb2c5f67","Type":"ContainerStarted","Data":"a4710a122ebdb206363fe15588e3fe7cc2c4c560ea2e2a0729a74a2f2620a850"}
Jul 17 14:23:08.316538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:08.316497    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" podStartSLOduration=16.316483373 podStartE2EDuration="16.316483373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:23:08.315378705 +0000 UTC m=+2266.702726146" watchObservedRunningTime="2026-07-17 14:23:08.316483373 +0000 UTC m=+2266.703830812"
Jul 17 14:23:09.304775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:09.304742    2575 generic.go:356] "Generic (PLEG): container finished" podID="00374493-02dd-4e29-9348-150fbb2c5f67" containerID="a4710a122ebdb206363fe15588e3fe7cc2c4c560ea2e2a0729a74a2f2620a850" exitCode=6
Jul 17 14:23:09.305214 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:09.304810    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" event={"ID":"00374493-02dd-4e29-9348-150fbb2c5f67","Type":"ContainerDied","Data":"a4710a122ebdb206363fe15588e3fe7cc2c4c560ea2e2a0729a74a2f2620a850"}
Jul 17 14:23:09.305214 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:09.304892    2575 scope.go:117] "RemoveContainer" containerID="1c783d22b5eebc6d6b7f4e3d2e0fc8003106cf0690d215a45bd0d6a20284c14c"
Jul 17 14:23:09.305214 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:09.305137    2575 scope.go:117] "RemoveContainer" containerID="a4710a122ebdb206363fe15588e3fe7cc2c4c560ea2e2a0729a74a2f2620a850"
Jul 17 14:23:09.305478 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:09.305368    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt_odh-ai-gateway-infra(00374493-02dd-4e29-9348-150fbb2c5f67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" podUID="00374493-02dd-4e29-9348-150fbb2c5f67"
Jul 17 14:23:09.339804 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:09.339779    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"]
Jul 17 14:23:10.444031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:10.444007    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"
Jul 17 14:23:10.587769 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:10.587693    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00374493-02dd-4e29-9348-150fbb2c5f67-maas-api-service-ca\") pod \"00374493-02dd-4e29-9348-150fbb2c5f67\" (UID: \"00374493-02dd-4e29-9348-150fbb2c5f67\") "
Jul 17 14:23:10.588055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:10.588034    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00374493-02dd-4e29-9348-150fbb2c5f67-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00374493-02dd-4e29-9348-150fbb2c5f67" (UID: "00374493-02dd-4e29-9348-150fbb2c5f67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:23:10.688433 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:10.688399    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00374493-02dd-4e29-9348-150fbb2c5f67-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:23:11.069438 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:11.069407    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:23:11.069612 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:11.069592    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:23:11.316807 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:11.316776    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"
Jul 17 14:23:11.316807 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:11.316803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt" event={"ID":"00374493-02dd-4e29-9348-150fbb2c5f67","Type":"ContainerDied","Data":"05807ef1c5360eb377790ebe8c2a2d571263defb0523ce6e18184edefe59ce2a"}
Jul 17 14:23:11.317009 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:11.316862    2575 scope.go:117] "RemoveContainer" containerID="a4710a122ebdb206363fe15588e3fe7cc2c4c560ea2e2a0729a74a2f2620a850"
Jul 17 14:23:11.346790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:11.346769    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"]
Jul 17 14:23:11.357856 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:11.357837    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8q8tt"]
Jul 17 14:23:12.074724 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:12.074690    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" path="/var/lib/kubelet/pods/00374493-02dd-4e29-9348-150fbb2c5f67/volumes"
Jul 17 14:23:23.069425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:23.069396    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:23:23.069824 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:23.069619    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:23:38.068950 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:38.068924    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:23:38.069396 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:38.069111    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:23:42.888491 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.888452    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"]
Jul 17 14:23:42.889041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889018    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889043    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889056    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889064    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889082    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889091    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889104    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889113    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889124    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.889540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889133    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.889540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889145    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.889540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889153    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.889540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889251    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889263    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.889540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889273    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:23:42.889540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.889308    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:23:42.892627 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.892601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"
Jul 17 14:23:42.894928 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.894908    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:23:42.900575 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.900009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"]
Jul 17 14:23:42.952521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:42.952494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fcc00d-b482-46e5-af07-774172e78d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw\" (UID: \"e9fcc00d-b482-46e5-af07-774172e78d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"
Jul 17 14:23:43.053946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:43.053908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fcc00d-b482-46e5-af07-774172e78d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw\" (UID: \"e9fcc00d-b482-46e5-af07-774172e78d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"
Jul 17 14:23:43.054525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:43.054504    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fcc00d-b482-46e5-af07-774172e78d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw\" (UID: \"e9fcc00d-b482-46e5-af07-774172e78d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"
Jul 17 14:23:43.203867 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:43.203789    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"
Jul 17 14:23:43.331477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:43.331452    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"]
Jul 17 14:23:43.333173 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:23:43.333142    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fcc00d_b482_46e5_af07_774172e78d2c.slice/crio-1daedc3098a9d26e00bd0cf0bb0a617d471ff9eaa43613c78df54333d343937e WatchSource:0}: Error finding container 1daedc3098a9d26e00bd0cf0bb0a617d471ff9eaa43613c78df54333d343937e: Status 404 returned error can't find the container with id 1daedc3098a9d26e00bd0cf0bb0a617d471ff9eaa43613c78df54333d343937e
Jul 17 14:23:43.472347 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:43.472313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" event={"ID":"e9fcc00d-b482-46e5-af07-774172e78d2c","Type":"ContainerStarted","Data":"1daedc3098a9d26e00bd0cf0bb0a617d471ff9eaa43613c78df54333d343937e"}
Jul 17 14:23:44.478316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:44.478269    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerID="7f8ef348af80fcfcf410cf57f28a282617188b2bb5a96fbb9267cfebf298e65b" exitCode=6
Jul 17 14:23:44.478670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:44.478371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" event={"ID":"e9fcc00d-b482-46e5-af07-774172e78d2c","Type":"ContainerDied","Data":"7f8ef348af80fcfcf410cf57f28a282617188b2bb5a96fbb9267cfebf298e65b"}
Jul 17 14:23:44.478670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:44.478496    2575 scope.go:117] "RemoveContainer" containerID="7f8ef348af80fcfcf410cf57f28a282617188b2bb5a96fbb9267cfebf298e65b"
Jul 17 14:23:45.484938 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:45.484900    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerID="37278b921b930d9d35048e647a518ddd21a98f2a89ff0e9c5d7067912b4d003d" exitCode=6
Jul 17 14:23:45.485339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:45.484984    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" event={"ID":"e9fcc00d-b482-46e5-af07-774172e78d2c","Type":"ContainerDied","Data":"37278b921b930d9d35048e647a518ddd21a98f2a89ff0e9c5d7067912b4d003d"}
Jul 17 14:23:45.485339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:45.485041    2575 scope.go:117] "RemoveContainer" containerID="7f8ef348af80fcfcf410cf57f28a282617188b2bb5a96fbb9267cfebf298e65b"
Jul 17 14:23:45.485339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:45.485199    2575 scope.go:117] "RemoveContainer" containerID="37278b921b930d9d35048e647a518ddd21a98f2a89ff0e9c5d7067912b4d003d"
Jul 17 14:23:45.485482 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:45.485463    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw_odh-ai-gateway-infra(e9fcc00d-b482-46e5-af07-774172e78d2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c"
Jul 17 14:23:46.491363 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:46.491336    2575 scope.go:117] "RemoveContainer" containerID="37278b921b930d9d35048e647a518ddd21a98f2a89ff0e9c5d7067912b4d003d"
Jul 17 14:23:46.491704 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:46.491537    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw_odh-ai-gateway-infra(e9fcc00d-b482-46e5-af07-774172e78d2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c"
Jul 17 14:23:49.068981 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:49.068947    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:23:49.069454 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:23:49.069158    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:23:59.069064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:59.069032    2575 scope.go:117] "RemoveContainer" containerID="37278b921b930d9d35048e647a518ddd21a98f2a89ff0e9c5d7067912b4d003d"
Jul 17 14:23:59.554229 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:59.554198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" event={"ID":"e9fcc00d-b482-46e5-af07-774172e78d2c","Type":"ContainerStarted","Data":"dede97d8eca355249de040e91e435cbfbb6d14232d538516d823d38fc7a32a89"}
Jul 17 14:23:59.573941 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:23:59.573899    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" podStartSLOduration=17.573885678 podStartE2EDuration="17.573885678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:23:59.571190865 +0000 UTC m=+2317.958538315" watchObservedRunningTime="2026-07-17 14:23:59.573885678 +0000 UTC m=+2317.961233181"
Jul 17 14:24:00.560797 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:00.560765    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerID="dede97d8eca355249de040e91e435cbfbb6d14232d538516d823d38fc7a32a89" exitCode=6
Jul 17 14:24:00.561293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:00.560849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" event={"ID":"e9fcc00d-b482-46e5-af07-774172e78d2c","Type":"ContainerDied","Data":"dede97d8eca355249de040e91e435cbfbb6d14232d538516d823d38fc7a32a89"}
Jul 17 14:24:00.561293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:00.560908    2575 scope.go:117] "RemoveContainer" containerID="37278b921b930d9d35048e647a518ddd21a98f2a89ff0e9c5d7067912b4d003d"
Jul 17 14:24:00.561293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:00.561175    2575 scope.go:117] "RemoveContainer" containerID="dede97d8eca355249de040e91e435cbfbb6d14232d538516d823d38fc7a32a89"
Jul 17 14:24:00.561482 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:00.561400    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw_odh-ai-gateway-infra(e9fcc00d-b482-46e5-af07-774172e78d2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c"
Jul 17 14:24:00.588582 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:00.588556    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"]
Jul 17 14:24:01.069149 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:01.069114    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:24:01.069374 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:01.069350    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:24:01.696192 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:01.696167    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"
Jul 17 14:24:01.810564 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:01.810539    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fcc00d-b482-46e5-af07-774172e78d2c-maas-api-service-ca\") pod \"e9fcc00d-b482-46e5-af07-774172e78d2c\" (UID: \"e9fcc00d-b482-46e5-af07-774172e78d2c\") "
Jul 17 14:24:01.810859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:01.810840    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fcc00d-b482-46e5-af07-774172e78d2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9fcc00d-b482-46e5-af07-774172e78d2c" (UID: "e9fcc00d-b482-46e5-af07-774172e78d2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:24:01.911415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:01.911353    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fcc00d-b482-46e5-af07-774172e78d2c-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:24:02.573134 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:02.573106    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"
Jul 17 14:24:02.573134 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:02.573117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw" event={"ID":"e9fcc00d-b482-46e5-af07-774172e78d2c","Type":"ContainerDied","Data":"1daedc3098a9d26e00bd0cf0bb0a617d471ff9eaa43613c78df54333d343937e"}
Jul 17 14:24:02.573433 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:02.573173    2575 scope.go:117] "RemoveContainer" containerID="dede97d8eca355249de040e91e435cbfbb6d14232d538516d823d38fc7a32a89"
Jul 17 14:24:02.590226 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:02.590201    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"]
Jul 17 14:24:02.595862 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:02.595838    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-fdvxw"]
Jul 17 14:24:04.074896 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:04.074864    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" path="/var/lib/kubelet/pods/e9fcc00d-b482-46e5-af07-774172e78d2c/volumes"
Jul 17 14:24:12.072106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:12.072075    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:24:12.072534 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:12.072269    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:24:24.069870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:24.069837    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:24:24.070414 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:24.070022    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:24:39.068956 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:39.068920    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:24:39.069502 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:39.069131    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:24:50.069662 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:50.069630    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:24:50.070084 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:50.069901    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 14:24:50.797713 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:50.797630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerStarted","Data":"ee8dc501372c44c1ad0657dcbfa9008ae6751999e4afdc65d6f642b0ee41502b"}
Jul 17 14:24:50.797859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:50.797841    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:24:51.803961 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:51.803929    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerID="ee8dc501372c44c1ad0657dcbfa9008ae6751999e4afdc65d6f642b0ee41502b" exitCode=1
Jul 17 14:24:51.804377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:51.804014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"ee8dc501372c44c1ad0657dcbfa9008ae6751999e4afdc65d6f642b0ee41502b"}
Jul 17 14:24:51.804377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:51.804077    2575 scope.go:117] "RemoveContainer" containerID="e722dd2f9fdb7bfa8c3bd2ded5fa7031ece8dd7d8a9af1bcee6fa72ef4d1b057"
Jul 17 14:24:51.804377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:51.804321    2575 scope.go:117] "RemoveContainer" containerID="ee8dc501372c44c1ad0657dcbfa9008ae6751999e4afdc65d6f642b0ee41502b"
Jul 17 14:24:51.804545 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:51.804516    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:24:52.809438 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:52.809404    2575 scope.go:117] "RemoveContainer" containerID="ee8dc501372c44c1ad0657dcbfa9008ae6751999e4afdc65d6f642b0ee41502b"
Jul 17 14:24:52.809812 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:52.809573    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:24:56.157175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:56.157140    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:24:56.157681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:24:56.157662    2575 scope.go:117] "RemoveContainer" containerID="ee8dc501372c44c1ad0657dcbfa9008ae6751999e4afdc65d6f642b0ee41502b"
Jul 17 14:24:56.157921 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:24:56.157896    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd_odh-ai-gateway-infra(c7cdf28e-11ac-4a4b-a502-50f89acc7093)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093"
Jul 17 14:25:06.637247 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.637217    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"]
Jul 17 14:25:06.769610 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.769584    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:25:06.851923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.851859    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdf28e-11ac-4a4b-a502-50f89acc7093-openshift-service-ca\") pod \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") "
Jul 17 14:25:06.851923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.851899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cdf28e-11ac-4a4b-a502-50f89acc7093-maas-api-tls\") pod \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") "
Jul 17 14:25:06.852088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.851925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtmm9\" (UniqueName: \"kubernetes.io/projected/c7cdf28e-11ac-4a4b-a502-50f89acc7093-kube-api-access-rtmm9\") pod \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\" (UID: \"c7cdf28e-11ac-4a4b-a502-50f89acc7093\") "
Jul 17 14:25:06.852253 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.852231    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cdf28e-11ac-4a4b-a502-50f89acc7093-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7cdf28e-11ac-4a4b-a502-50f89acc7093" (UID: "c7cdf28e-11ac-4a4b-a502-50f89acc7093"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:25:06.854010 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.853987    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7cdf28e-11ac-4a4b-a502-50f89acc7093-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7cdf28e-11ac-4a4b-a502-50f89acc7093" (UID: "c7cdf28e-11ac-4a4b-a502-50f89acc7093"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:25:06.854078 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.854033    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cdf28e-11ac-4a4b-a502-50f89acc7093-kube-api-access-rtmm9" (OuterVolumeSpecName: "kube-api-access-rtmm9") pod "c7cdf28e-11ac-4a4b-a502-50f89acc7093" (UID: "c7cdf28e-11ac-4a4b-a502-50f89acc7093"). InnerVolumeSpecName "kube-api-access-rtmm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:25:06.875957 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.875932    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd" event={"ID":"c7cdf28e-11ac-4a4b-a502-50f89acc7093","Type":"ContainerDied","Data":"6bbf44ec9ae818464a4ff2e8fe39e2897abda981a1ed03936e1f7bd6acbfc366"}
Jul 17 14:25:06.876037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.875984    2575 scope.go:117] "RemoveContainer" containerID="ee8dc501372c44c1ad0657dcbfa9008ae6751999e4afdc65d6f642b0ee41502b"
Jul 17 14:25:06.876037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.875941    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"
Jul 17 14:25:06.898087 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.898063    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"]
Jul 17 14:25:06.900290 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.900252    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7874222-9b6f8dfb7-fx5wd"]
Jul 17 14:25:06.953261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.953236    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdf28e-11ac-4a4b-a502-50f89acc7093-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:25:06.953419 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.953401    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cdf28e-11ac-4a4b-a502-50f89acc7093-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:25:06.953516 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:06.953504    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmm9\" (UniqueName: \"kubernetes.io/projected/c7cdf28e-11ac-4a4b-a502-50f89acc7093-kube-api-access-rtmm9\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:25:08.076117 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:08.076087    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" path="/var/lib/kubelet/pods/c7cdf28e-11ac-4a4b-a502-50f89acc7093/volumes"
Jul 17 14:25:14.443339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443305    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"]
Jul 17 14:25:14.443736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443698    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443708    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443719    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443724    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443732    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443738    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443744    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443749    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443773    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443778    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443786    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443791    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443799    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443804    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443817    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443822    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443876    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443882    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443887    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443892    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443896    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443904    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443911    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00374493-02dd-4e29-9348-150fbb2c5f67" containerName="revoke-keys"
Jul 17 14:25:14.443923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.443916    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdefe6f5-3cf9-4776-ac3e-1c9a3c244212" containerName="revoke-keys"
Jul 17 14:25:14.448472 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.448451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.451948 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.451928    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-73fe1bea-serving-cert\""
Jul 17 14:25:14.458412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.458388    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"]
Jul 17 14:25:14.614788 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.614758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.614958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.614805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57qc\" (UniqueName: \"kubernetes.io/projected/534a979a-fa8e-4b5e-b3d5-4d2835152f86-kube-api-access-t57qc\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.614958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.614824    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a979a-fa8e-4b5e-b3d5-4d2835152f86-openshift-service-ca\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.715802 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.715776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57qc\" (UniqueName: \"kubernetes.io/projected/534a979a-fa8e-4b5e-b3d5-4d2835152f86-kube-api-access-t57qc\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.715923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.715808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a979a-fa8e-4b5e-b3d5-4d2835152f86-openshift-service-ca\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.715923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.715865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.715995 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:14.715953    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-serving-cert: secret "maas-api-e2e-derive-73fe1bea-serving-cert" not found
Jul 17 14:25:14.716029 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:14.716011    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls podName:534a979a-fa8e-4b5e-b3d5-4d2835152f86 nodeName:}" failed. No retries permitted until 2026-07-17 14:25:15.215997721 +0000 UTC m=+2393.603345142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls") pod "maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" (UID: "534a979a-fa8e-4b5e-b3d5-4d2835152f86") : secret "maas-api-e2e-derive-73fe1bea-serving-cert" not found
Jul 17 14:25:14.716512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.716495    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a979a-fa8e-4b5e-b3d5-4d2835152f86-openshift-service-ca\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:14.726609 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:14.726588    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57qc\" (UniqueName: \"kubernetes.io/projected/534a979a-fa8e-4b5e-b3d5-4d2835152f86-kube-api-access-t57qc\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:15.220926 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.220893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:15.223216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.223193    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls\") pod \"maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:15.358418 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.358384    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:15.489788 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.489762    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"]
Jul 17 14:25:15.492062 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:25:15.492036    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534a979a_fa8e_4b5e_b3d5_4d2835152f86.slice/crio-471ed54e144a41fa25ac28a5e66efb34890e2bf14e6ab80fc1baf7944d8448c8 WatchSource:0}: Error finding container 471ed54e144a41fa25ac28a5e66efb34890e2bf14e6ab80fc1baf7944d8448c8: Status 404 returned error can't find the container with id 471ed54e144a41fa25ac28a5e66efb34890e2bf14e6ab80fc1baf7944d8448c8
Jul 17 14:25:15.919164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.919134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerStarted","Data":"ab6a13110a657aaf61c39de2421b171034b825281c692bc351bfed7e3f4a08ba"}
Jul 17 14:25:15.919321 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.919180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerStarted","Data":"471ed54e144a41fa25ac28a5e66efb34890e2bf14e6ab80fc1baf7944d8448c8"}
Jul 17 14:25:15.919321 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.919224    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:15.936767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:15.936723    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podStartSLOduration=1.627158121 podStartE2EDuration="1.936710317s" totalImagesPullingTime="309.552196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:25:15.93464967 +0000 UTC m=+2394.321997109" watchObservedRunningTime="2026-07-17 14:25:15.936710317 +0000 UTC m=+2394.324057756"
Jul 17 14:25:16.925649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:16.925607    2575 generic.go:356] "Generic (PLEG): container finished" podID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerID="ab6a13110a657aaf61c39de2421b171034b825281c692bc351bfed7e3f4a08ba" exitCode=1
Jul 17 14:25:16.926074 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:16.925689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerDied","Data":"ab6a13110a657aaf61c39de2421b171034b825281c692bc351bfed7e3f4a08ba"}
Jul 17 14:25:16.926074 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:16.925940    2575 scope.go:117] "RemoveContainer" containerID="ab6a13110a657aaf61c39de2421b171034b825281c692bc351bfed7e3f4a08ba"
Jul 17 14:25:17.932042 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:17.931959    2575 generic.go:356] "Generic (PLEG): container finished" podID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerID="bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e" exitCode=1
Jul 17 14:25:17.932042 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:17.932036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerDied","Data":"bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e"}
Jul 17 14:25:17.932497 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:17.932083    2575 scope.go:117] "RemoveContainer" containerID="ab6a13110a657aaf61c39de2421b171034b825281c692bc351bfed7e3f4a08ba"
Jul 17 14:25:17.932497 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:17.932339    2575 scope.go:117] "RemoveContainer" containerID="bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e"
Jul 17 14:25:17.932570 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:17.932539    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:18.938734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:18.938709    2575 scope.go:117] "RemoveContainer" containerID="bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e"
Jul 17 14:25:18.939113 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:18.938893    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:20.919748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:20.919718    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:20.920140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:20.920068    2575 scope.go:117] "RemoveContainer" containerID="bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e"
Jul 17 14:25:20.920243 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:20.920224    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:22.164442 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:22.164411    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:25:22.167946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:22.167925    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:25:22.172331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:22.172313    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:25:22.174307 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:22.174267    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:25:22.175830 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:22.175814    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:25:22.181971 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:22.181956    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:25:25.359527 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:25.359494    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:25.359907 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:25.359878    2575 scope.go:117] "RemoveContainer" containerID="bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e"
Jul 17 14:25:25.360071 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:25.360043    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:40.069642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:40.069614    2575 scope.go:117] "RemoveContainer" containerID="bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e"
Jul 17 14:25:41.042941 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:41.042858    2575 generic.go:356] "Generic (PLEG): container finished" podID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerID="a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85" exitCode=1
Jul 17 14:25:41.043077 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:41.042932    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerDied","Data":"a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85"}
Jul 17 14:25:41.043077 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:41.042987    2575 scope.go:117] "RemoveContainer" containerID="bbcd2fe3d377f2c47392c2d94833ea545ddc345a3affa0cc10902aae21283c2e"
Jul 17 14:25:41.043375 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:41.043353    2575 scope.go:117] "RemoveContainer" containerID="a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85"
Jul 17 14:25:41.043599 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:41.043579    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:42.103489 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.103455    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"]
Jul 17 14:25:42.103853 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.103835    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:42.103853 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.103848    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:42.103944 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.103932    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:42.103986 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.103951    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:42.103986 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.103958    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:25:42.103986 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.103964    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9fcc00d-b482-46e5-af07-774172e78d2c" containerName="revoke-keys"
Jul 17 14:25:42.108138 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.108123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"
Jul 17 14:25:42.110334 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.110313    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:25:42.113413 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.113390    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"]
Jul 17 14:25:42.148864 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.148842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f880d95-4496-4f38-b7a1-1e4c442e3974-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s\" (UID: \"6f880d95-4496-4f38-b7a1-1e4c442e3974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"
Jul 17 14:25:42.250198 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.250175    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f880d95-4496-4f38-b7a1-1e4c442e3974-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s\" (UID: \"6f880d95-4496-4f38-b7a1-1e4c442e3974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"
Jul 17 14:25:42.250763 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.250747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f880d95-4496-4f38-b7a1-1e4c442e3974-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s\" (UID: \"6f880d95-4496-4f38-b7a1-1e4c442e3974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"
Jul 17 14:25:42.420185 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.420118    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"
Jul 17 14:25:42.547954 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:42.547928    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"]
Jul 17 14:25:42.550785 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:25:42.550758    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f880d95_4496_4f38_b7a1_1e4c442e3974.slice/crio-bd3f27abfdf3eb8e57a7785149b09de2d8aed9d67378d50cd287a17d0f39b57e WatchSource:0}: Error finding container bd3f27abfdf3eb8e57a7785149b09de2d8aed9d67378d50cd287a17d0f39b57e: Status 404 returned error can't find the container with id bd3f27abfdf3eb8e57a7785149b09de2d8aed9d67378d50cd287a17d0f39b57e
Jul 17 14:25:43.054525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:43.054460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" event={"ID":"6f880d95-4496-4f38-b7a1-1e4c442e3974","Type":"ContainerStarted","Data":"1449c295d7df8ef7c5f44ace22c7b65bbd8de6603c66a635c979c32622003465"}
Jul 17 14:25:43.054525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:43.054507    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" event={"ID":"6f880d95-4496-4f38-b7a1-1e4c442e3974","Type":"ContainerStarted","Data":"bd3f27abfdf3eb8e57a7785149b09de2d8aed9d67378d50cd287a17d0f39b57e"}
Jul 17 14:25:43.072987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:43.072945    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" podStartSLOduration=1.072931291 podStartE2EDuration="1.072931291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:25:43.071809283 +0000 UTC m=+2421.459156722" watchObservedRunningTime="2026-07-17 14:25:43.072931291 +0000 UTC m=+2421.460278731"
Jul 17 14:25:44.060849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:44.060811    2575 generic.go:356] "Generic (PLEG): container finished" podID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerID="1449c295d7df8ef7c5f44ace22c7b65bbd8de6603c66a635c979c32622003465" exitCode=6
Jul 17 14:25:44.061327 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:44.060901    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" event={"ID":"6f880d95-4496-4f38-b7a1-1e4c442e3974","Type":"ContainerDied","Data":"1449c295d7df8ef7c5f44ace22c7b65bbd8de6603c66a635c979c32622003465"}
Jul 17 14:25:44.061327 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:44.061064    2575 scope.go:117] "RemoveContainer" containerID="1449c295d7df8ef7c5f44ace22c7b65bbd8de6603c66a635c979c32622003465"
Jul 17 14:25:45.067347 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:45.067315    2575 generic.go:356] "Generic (PLEG): container finished" podID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerID="0a2ad69643f01779151fe40ce457d9b950c458f954f7ba53d1111b22a6ef0cf8" exitCode=6
Jul 17 14:25:45.067748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:45.067406    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" event={"ID":"6f880d95-4496-4f38-b7a1-1e4c442e3974","Type":"ContainerDied","Data":"0a2ad69643f01779151fe40ce457d9b950c458f954f7ba53d1111b22a6ef0cf8"}
Jul 17 14:25:45.067748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:45.067463    2575 scope.go:117] "RemoveContainer" containerID="1449c295d7df8ef7c5f44ace22c7b65bbd8de6603c66a635c979c32622003465"
Jul 17 14:25:45.067748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:45.067639    2575 scope.go:117] "RemoveContainer" containerID="0a2ad69643f01779151fe40ce457d9b950c458f954f7ba53d1111b22a6ef0cf8"
Jul 17 14:25:45.067879 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:45.067833    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s_odh-ai-gateway-infra(6f880d95-4496-4f38-b7a1-1e4c442e3974)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974"
Jul 17 14:25:45.359227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:45.359161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:45.359558 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:45.359543    2575 scope.go:117] "RemoveContainer" containerID="a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85"
Jul 17 14:25:45.359720 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:45.359702    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:45.919958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:45.919925    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:25:46.080344 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:46.080313    2575 scope.go:117] "RemoveContainer" containerID="a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85"
Jul 17 14:25:46.080823 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:46.080424    2575 scope.go:117] "RemoveContainer" containerID="0a2ad69643f01779151fe40ce457d9b950c458f954f7ba53d1111b22a6ef0cf8"
Jul 17 14:25:46.080823 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:46.080463    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:46.080823 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:46.080619    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s_odh-ai-gateway-infra(6f880d95-4496-4f38-b7a1-1e4c442e3974)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974"
Jul 17 14:25:57.069648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:57.069620    2575 scope.go:117] "RemoveContainer" containerID="0a2ad69643f01779151fe40ce457d9b950c458f954f7ba53d1111b22a6ef0cf8"
Jul 17 14:25:58.135479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:58.135411    2575 generic.go:356] "Generic (PLEG): container finished" podID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerID="7ec3be902eb12e6b400ef83191f4fc2b224437903979dcfa0509adf5e2771354" exitCode=6
Jul 17 14:25:58.135873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:58.135488    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" event={"ID":"6f880d95-4496-4f38-b7a1-1e4c442e3974","Type":"ContainerDied","Data":"7ec3be902eb12e6b400ef83191f4fc2b224437903979dcfa0509adf5e2771354"}
Jul 17 14:25:58.135873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:58.135561    2575 scope.go:117] "RemoveContainer" containerID="0a2ad69643f01779151fe40ce457d9b950c458f954f7ba53d1111b22a6ef0cf8"
Jul 17 14:25:58.135873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:58.135822    2575 scope.go:117] "RemoveContainer" containerID="7ec3be902eb12e6b400ef83191f4fc2b224437903979dcfa0509adf5e2771354"
Jul 17 14:25:58.136022 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:58.135990    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s_odh-ai-gateway-infra(6f880d95-4496-4f38-b7a1-1e4c442e3974)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974"
Jul 17 14:25:59.069107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:59.069082    2575 scope.go:117] "RemoveContainer" containerID="a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85"
Jul 17 14:25:59.069261 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:25:59.069234    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:25:59.168749 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:59.168721    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"]
Jul 17 14:25:59.297039 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:59.297017    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"
Jul 17 14:25:59.394507 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:59.394434    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f880d95-4496-4f38-b7a1-1e4c442e3974-maas-api-service-ca\") pod \"6f880d95-4496-4f38-b7a1-1e4c442e3974\" (UID: \"6f880d95-4496-4f38-b7a1-1e4c442e3974\") "
Jul 17 14:25:59.394752 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:59.394731    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f880d95-4496-4f38-b7a1-1e4c442e3974-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f880d95-4496-4f38-b7a1-1e4c442e3974" (UID: "6f880d95-4496-4f38-b7a1-1e4c442e3974"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:25:59.495652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:25:59.495623    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f880d95-4496-4f38-b7a1-1e4c442e3974-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:26:00.146580 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:00.146554    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"
Jul 17 14:26:00.146729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:00.146577    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s" event={"ID":"6f880d95-4496-4f38-b7a1-1e4c442e3974","Type":"ContainerDied","Data":"bd3f27abfdf3eb8e57a7785149b09de2d8aed9d67378d50cd287a17d0f39b57e"}
Jul 17 14:26:00.146729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:00.146637    2575 scope.go:117] "RemoveContainer" containerID="7ec3be902eb12e6b400ef83191f4fc2b224437903979dcfa0509adf5e2771354"
Jul 17 14:26:00.164561 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:00.164539    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"]
Jul 17 14:26:00.169175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:00.169155    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-98c4s"]
Jul 17 14:26:02.074285 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:02.074238    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" path="/var/lib/kubelet/pods/6f880d95-4496-4f38-b7a1-1e4c442e3974/volumes"
Jul 17 14:26:10.069342 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:10.069307    2575 scope.go:117] "RemoveContainer" containerID="a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85"
Jul 17 14:26:11.199219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:11.199190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerStarted","Data":"2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"}
Jul 17 14:26:11.199654 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:11.199414    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:26:12.205210 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:12.205181    2575 generic.go:356] "Generic (PLEG): container finished" podID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094" exitCode=1
Jul 17 14:26:12.205621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:12.205262    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerDied","Data":"2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"}
Jul 17 14:26:12.205621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:12.205342    2575 scope.go:117] "RemoveContainer" containerID="a0ce9f9230767b638a890ca029b4a549bcd1dd0ed1531a49e3c7fa05a398fa85"
Jul 17 14:26:12.205621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:12.205428    2575 scope.go:117] "RemoveContainer" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"
Jul 17 14:26:12.205778 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:12.205628    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:26:13.216485 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:13.216460    2575 scope.go:117] "RemoveContainer" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"
Jul 17 14:26:13.216895 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:13.216615    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:26:15.359392 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:15.359351    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:26:15.359848 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:15.359703    2575 scope.go:117] "RemoveContainer" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"
Jul 17 14:26:15.359914 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:15.359894    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:26:28.069157 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:28.069128    2575 scope.go:117] "RemoveContainer" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"
Jul 17 14:26:28.069579 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:28.069309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:26:33.029163 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029132    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"]
Jul 17 14:26:33.029551 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029525    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:26:33.029551 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029536    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7cdf28e-11ac-4a4b-a502-50f89acc7093" containerName="maas-api"
Jul 17 14:26:33.029631 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029556    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:26:33.029631 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029562    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:26:33.029631 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029569    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:26:33.029631 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029575    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:26:33.029750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029649    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:26:33.029750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.029663    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:26:33.032602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.032585    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"
Jul 17 14:26:33.037681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.037654    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:26:33.043540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.043519    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"]
Jul 17 14:26:33.052675 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.052645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558951a-357f-4a5b-ad8e-3be6c620b798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-bht9h\" (UID: \"6558951a-357f-4a5b-ad8e-3be6c620b798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"
Jul 17 14:26:33.153247 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.153221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558951a-357f-4a5b-ad8e-3be6c620b798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-bht9h\" (UID: \"6558951a-357f-4a5b-ad8e-3be6c620b798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"
Jul 17 14:26:33.153788 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.153770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558951a-357f-4a5b-ad8e-3be6c620b798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-bht9h\" (UID: \"6558951a-357f-4a5b-ad8e-3be6c620b798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"
Jul 17 14:26:33.342698 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.342640    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"
Jul 17 14:26:33.470221 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:33.470065    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"]
Jul 17 14:26:33.472399 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:26:33.472369    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6558951a_357f_4a5b_ad8e_3be6c620b798.slice/crio-0c053e568195c5714a0dfedf83aefc6824b7e9a7fe6e1cb72b084aff1fce92bc WatchSource:0}: Error finding container 0c053e568195c5714a0dfedf83aefc6824b7e9a7fe6e1cb72b084aff1fce92bc: Status 404 returned error can't find the container with id 0c053e568195c5714a0dfedf83aefc6824b7e9a7fe6e1cb72b084aff1fce92bc
Jul 17 14:26:34.331040 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:34.331009    2575 generic.go:356] "Generic (PLEG): container finished" podID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerID="9fd083a4e96b7ae71a8523a36d2868e45cd680759d56b2502a8e292139149607" exitCode=6
Jul 17 14:26:34.331432 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:34.331092    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" event={"ID":"6558951a-357f-4a5b-ad8e-3be6c620b798","Type":"ContainerDied","Data":"9fd083a4e96b7ae71a8523a36d2868e45cd680759d56b2502a8e292139149607"}
Jul 17 14:26:34.331432 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:34.331135    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" event={"ID":"6558951a-357f-4a5b-ad8e-3be6c620b798","Type":"ContainerStarted","Data":"0c053e568195c5714a0dfedf83aefc6824b7e9a7fe6e1cb72b084aff1fce92bc"}
Jul 17 14:26:34.331432 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:34.331297    2575 scope.go:117] "RemoveContainer" containerID="9fd083a4e96b7ae71a8523a36d2868e45cd680759d56b2502a8e292139149607"
Jul 17 14:26:35.336647 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:35.336618    2575 generic.go:356] "Generic (PLEG): container finished" podID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerID="f8f4d0e652b14fea5476943270b9942dc2c18d21db8b88cf82ece6a51589cc98" exitCode=6
Jul 17 14:26:35.337080 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:35.336703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" event={"ID":"6558951a-357f-4a5b-ad8e-3be6c620b798","Type":"ContainerDied","Data":"f8f4d0e652b14fea5476943270b9942dc2c18d21db8b88cf82ece6a51589cc98"}
Jul 17 14:26:35.337080 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:35.336755    2575 scope.go:117] "RemoveContainer" containerID="9fd083a4e96b7ae71a8523a36d2868e45cd680759d56b2502a8e292139149607"
Jul 17 14:26:35.337080 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:35.336959    2575 scope.go:117] "RemoveContainer" containerID="f8f4d0e652b14fea5476943270b9942dc2c18d21db8b88cf82ece6a51589cc98"
Jul 17 14:26:35.337244 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:35.337183    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-bht9h_odh-ai-gateway-infra(6558951a-357f-4a5b-ad8e-3be6c620b798)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798"
Jul 17 14:26:36.343179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:36.343151    2575 scope.go:117] "RemoveContainer" containerID="f8f4d0e652b14fea5476943270b9942dc2c18d21db8b88cf82ece6a51589cc98"
Jul 17 14:26:36.343568 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:36.343322    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-bht9h_odh-ai-gateway-infra(6558951a-357f-4a5b-ad8e-3be6c620b798)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798"
Jul 17 14:26:42.071574 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:42.071542    2575 scope.go:117] "RemoveContainer" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"
Jul 17 14:26:42.072025 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:42.071731    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:26:50.069054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:50.069028    2575 scope.go:117] "RemoveContainer" containerID="f8f4d0e652b14fea5476943270b9942dc2c18d21db8b88cf82ece6a51589cc98"
Jul 17 14:26:50.412255 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:50.412189    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" event={"ID":"6558951a-357f-4a5b-ad8e-3be6c620b798","Type":"ContainerStarted","Data":"0d314e7f1e045dd9d1ab85f86bc778a711d1bc7e1857e21ff0fc90fc9d9ff513"}
Jul 17 14:26:50.429566 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:50.429513    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" podStartSLOduration=17.429497148 podStartE2EDuration="17.429497148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:26:50.427937733 +0000 UTC m=+2488.815285173" watchObservedRunningTime="2026-07-17 14:26:50.429497148 +0000 UTC m=+2488.816844587"
Jul 17 14:26:51.417901 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:51.417866    2575 generic.go:356] "Generic (PLEG): container finished" podID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerID="0d314e7f1e045dd9d1ab85f86bc778a711d1bc7e1857e21ff0fc90fc9d9ff513" exitCode=6
Jul 17 14:26:51.418299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:51.417914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" event={"ID":"6558951a-357f-4a5b-ad8e-3be6c620b798","Type":"ContainerDied","Data":"0d314e7f1e045dd9d1ab85f86bc778a711d1bc7e1857e21ff0fc90fc9d9ff513"}
Jul 17 14:26:51.418299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:51.417958    2575 scope.go:117] "RemoveContainer" containerID="f8f4d0e652b14fea5476943270b9942dc2c18d21db8b88cf82ece6a51589cc98"
Jul 17 14:26:51.418299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:51.418173    2575 scope.go:117] "RemoveContainer" containerID="0d314e7f1e045dd9d1ab85f86bc778a711d1bc7e1857e21ff0fc90fc9d9ff513"
Jul 17 14:26:51.418429 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:51.418357    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-bht9h_odh-ai-gateway-infra(6558951a-357f-4a5b-ad8e-3be6c620b798)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798"
Jul 17 14:26:51.445300 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:51.445251    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"]
Jul 17 14:26:52.553371 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:52.553349    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"
Jul 17 14:26:52.610632 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:52.610605    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558951a-357f-4a5b-ad8e-3be6c620b798-maas-api-service-ca\") pod \"6558951a-357f-4a5b-ad8e-3be6c620b798\" (UID: \"6558951a-357f-4a5b-ad8e-3be6c620b798\") "
Jul 17 14:26:52.610918 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:52.610897    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6558951a-357f-4a5b-ad8e-3be6c620b798-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6558951a-357f-4a5b-ad8e-3be6c620b798" (UID: "6558951a-357f-4a5b-ad8e-3be6c620b798"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:26:52.711205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:52.711144    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558951a-357f-4a5b-ad8e-3be6c620b798-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:26:53.429218 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:53.429187    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h" event={"ID":"6558951a-357f-4a5b-ad8e-3be6c620b798","Type":"ContainerDied","Data":"0c053e568195c5714a0dfedf83aefc6824b7e9a7fe6e1cb72b084aff1fce92bc"}
Jul 17 14:26:53.429405 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:53.429246    2575 scope.go:117] "RemoveContainer" containerID="0d314e7f1e045dd9d1ab85f86bc778a711d1bc7e1857e21ff0fc90fc9d9ff513"
Jul 17 14:26:53.429405 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:53.429195    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"
Jul 17 14:26:53.453131 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:53.453101    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"]
Jul 17 14:26:53.460866 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:53.460842    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-bht9h"]
Jul 17 14:26:54.074700 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:54.074672    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" path="/var/lib/kubelet/pods/6558951a-357f-4a5b-ad8e-3be6c620b798/volumes"
Jul 17 14:26:56.069620 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:56.069587    2575 scope.go:117] "RemoveContainer" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"
Jul 17 14:26:57.448923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:57.448893    2575 generic.go:356] "Generic (PLEG): container finished" podID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88" exitCode=1
Jul 17 14:26:57.449346 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:57.448941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerDied","Data":"e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"}
Jul 17 14:26:57.449346 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:57.448990    2575 scope.go:117] "RemoveContainer" containerID="2836820d82da0146711366cf9806c2c40e64a8c7f631df9aaff5d3112b8d6094"
Jul 17 14:26:57.449346 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:26:57.449260    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:26:57.449486 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:26:57.449466    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:27:01.199987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:01.199955    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:27:01.200394 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:01.200262    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:27:01.200442 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:01.200427    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:27:05.359108 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:05.359067    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:27:05.359485 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:05.359426    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:27:05.359602 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:05.359582    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:27:08.862228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862184    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"]
Jul 17 14:27:08.862764 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862744    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.862849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862767    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.862849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862781    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.862849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862789    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.862849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862802    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:27:08.862849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862811    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:27:08.862849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862820    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.862849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862828    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.863168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.862989    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.863168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.863003    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.863168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.863012    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6558951a-357f-4a5b-ad8e-3be6c620b798" containerName="revoke-keys"
Jul 17 14:27:08.863168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.863022    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f880d95-4496-4f38-b7a1-1e4c442e3974" containerName="revoke-keys"
Jul 17 14:27:08.866097 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.866077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"
Jul 17 14:27:08.868783 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.868765    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:27:08.879988 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:08.879964    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"]
Jul 17 14:27:09.034010 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.033980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7a78d-107a-428e-bfa4-c1f06b6ecebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d\" (UID: \"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"
Jul 17 14:27:09.135415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.135354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7a78d-107a-428e-bfa4-c1f06b6ecebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d\" (UID: \"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"
Jul 17 14:27:09.135936 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.135919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7a78d-107a-428e-bfa4-c1f06b6ecebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d\" (UID: \"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"
Jul 17 14:27:09.175467 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.175449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"
Jul 17 14:27:09.304130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.304102    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"]
Jul 17 14:27:09.305154 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:27:09.305129    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f7a78d_107a_428e_bfa4_c1f06b6ecebd.slice/crio-17f486b9dab8484a1bfb3fdff0369deb329d2313079e2bce30240a3836631530 WatchSource:0}: Error finding container 17f486b9dab8484a1bfb3fdff0369deb329d2313079e2bce30240a3836631530: Status 404 returned error can't find the container with id 17f486b9dab8484a1bfb3fdff0369deb329d2313079e2bce30240a3836631530
Jul 17 14:27:09.513958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.513930    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" event={"ID":"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd","Type":"ContainerStarted","Data":"e95da45f85c395e996a57ac31987856991c5d5e1a4fe32e3d6aea10ec752fa1d"}
Jul 17 14:27:09.514058 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.513986    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" event={"ID":"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd","Type":"ContainerStarted","Data":"17f486b9dab8484a1bfb3fdff0369deb329d2313079e2bce30240a3836631530"}
Jul 17 14:27:09.531300 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:09.531245    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" podStartSLOduration=1.5312301289999999 podStartE2EDuration="1.531230129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:27:09.528978334 +0000 UTC m=+2507.916325777" watchObservedRunningTime="2026-07-17 14:27:09.531230129 +0000 UTC m=+2507.918577567"
Jul 17 14:27:10.519593 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:10.519560    2575 generic.go:356] "Generic (PLEG): container finished" podID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerID="e95da45f85c395e996a57ac31987856991c5d5e1a4fe32e3d6aea10ec752fa1d" exitCode=7
Jul 17 14:27:10.520009 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:10.519607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" event={"ID":"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd","Type":"ContainerDied","Data":"e95da45f85c395e996a57ac31987856991c5d5e1a4fe32e3d6aea10ec752fa1d"}
Jul 17 14:27:10.520009 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:10.519815    2575 scope.go:117] "RemoveContainer" containerID="e95da45f85c395e996a57ac31987856991c5d5e1a4fe32e3d6aea10ec752fa1d"
Jul 17 14:27:11.494314 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.494269    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"]
Jul 17 14:27:11.497750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.497734    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"
Jul 17 14:27:11.505373 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.505348    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"]
Jul 17 14:27:11.526823 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.526798    2575 generic.go:356] "Generic (PLEG): container finished" podID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerID="9c55404efc0976ca0311581cf5f5f721571d618b500358569a75a33e0931754d" exitCode=7
Jul 17 14:27:11.527224 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.526851    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" event={"ID":"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd","Type":"ContainerDied","Data":"9c55404efc0976ca0311581cf5f5f721571d618b500358569a75a33e0931754d"}
Jul 17 14:27:11.527224 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.526909    2575 scope.go:117] "RemoveContainer" containerID="e95da45f85c395e996a57ac31987856991c5d5e1a4fe32e3d6aea10ec752fa1d"
Jul 17 14:27:11.527224 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.527081    2575 scope.go:117] "RemoveContainer" containerID="9c55404efc0976ca0311581cf5f5f721571d618b500358569a75a33e0931754d"
Jul 17 14:27:11.527415 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:11.527317    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d_odh-ai-gateway-infra(b9f7a78d-107a-428e-bfa4-c1f06b6ecebd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd"
Jul 17 14:27:11.654745 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.654718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d9bd34-0ad3-4e70-b528-c8d8f47fe42c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-gxbjq\" (UID: \"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"
Jul 17 14:27:11.755160 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.755103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d9bd34-0ad3-4e70-b528-c8d8f47fe42c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-gxbjq\" (UID: \"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"
Jul 17 14:27:11.755703 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.755686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d9bd34-0ad3-4e70-b528-c8d8f47fe42c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-gxbjq\" (UID: \"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"
Jul 17 14:27:11.809161 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.809139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"
Jul 17 14:27:11.938511 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:11.938377    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"]
Jul 17 14:27:11.941148 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:27:11.941120    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d9bd34_0ad3_4e70_b528_c8d8f47fe42c.slice/crio-75956740fd4addff9d13c32006643a7a8d0bf0b2f4e939098c89eb812d777243 WatchSource:0}: Error finding container 75956740fd4addff9d13c32006643a7a8d0bf0b2f4e939098c89eb812d777243: Status 404 returned error can't find the container with id 75956740fd4addff9d13c32006643a7a8d0bf0b2f4e939098c89eb812d777243
Jul 17 14:27:12.532933 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:12.532902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" event={"ID":"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c","Type":"ContainerStarted","Data":"084e0d7e1ac7461719525aa13913fbd03dfcad834b091460b9c4c090152c71b7"}
Jul 17 14:27:12.533335 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:12.532952    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" event={"ID":"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c","Type":"ContainerStarted","Data":"75956740fd4addff9d13c32006643a7a8d0bf0b2f4e939098c89eb812d777243"}
Jul 17 14:27:12.534591 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:12.534574    2575 scope.go:117] "RemoveContainer" containerID="9c55404efc0976ca0311581cf5f5f721571d618b500358569a75a33e0931754d"
Jul 17 14:27:12.534742 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:12.534722    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d_odh-ai-gateway-infra(b9f7a78d-107a-428e-bfa4-c1f06b6ecebd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd"
Jul 17 14:27:12.550320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:12.550264    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" podStartSLOduration=1.5502509070000001 podStartE2EDuration="1.550250907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:27:12.548761782 +0000 UTC m=+2510.936109222" watchObservedRunningTime="2026-07-17 14:27:12.550250907 +0000 UTC m=+2510.937598347"
Jul 17 14:27:13.540786 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:13.540753    2575 generic.go:356] "Generic (PLEG): container finished" podID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerID="084e0d7e1ac7461719525aa13913fbd03dfcad834b091460b9c4c090152c71b7" exitCode=6
Jul 17 14:27:13.541240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:13.540835    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" event={"ID":"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c","Type":"ContainerDied","Data":"084e0d7e1ac7461719525aa13913fbd03dfcad834b091460b9c4c090152c71b7"}
Jul 17 14:27:13.541240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:13.541016    2575 scope.go:117] "RemoveContainer" containerID="084e0d7e1ac7461719525aa13913fbd03dfcad834b091460b9c4c090152c71b7"
Jul 17 14:27:14.547163 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:14.547130    2575 generic.go:356] "Generic (PLEG): container finished" podID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerID="9f7ac78f8b1cd53954b86479639b556ac09860a5a2469a73ccbf345ddfa2a84d" exitCode=6
Jul 17 14:27:14.547501 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:14.547212    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" event={"ID":"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c","Type":"ContainerDied","Data":"9f7ac78f8b1cd53954b86479639b556ac09860a5a2469a73ccbf345ddfa2a84d"}
Jul 17 14:27:14.547501 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:14.547270    2575 scope.go:117] "RemoveContainer" containerID="084e0d7e1ac7461719525aa13913fbd03dfcad834b091460b9c4c090152c71b7"
Jul 17 14:27:14.547501 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:14.547474    2575 scope.go:117] "RemoveContainer" containerID="9f7ac78f8b1cd53954b86479639b556ac09860a5a2469a73ccbf345ddfa2a84d"
Jul 17 14:27:14.547672 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:14.547653    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-gxbjq_odh-ai-gateway-infra(92d9bd34-0ad3-4e70-b528-c8d8f47fe42c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c"
Jul 17 14:27:15.553155 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:15.553130    2575 scope.go:117] "RemoveContainer" containerID="9f7ac78f8b1cd53954b86479639b556ac09860a5a2469a73ccbf345ddfa2a84d"
Jul 17 14:27:15.553526 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:15.553314    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-gxbjq_odh-ai-gateway-infra(92d9bd34-0ad3-4e70-b528-c8d8f47fe42c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c"
Jul 17 14:27:18.069756 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:18.069729    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:27:18.070147 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:18.069946    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:27:22.190422 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.190389    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"]
Jul 17 14:27:22.195045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.195028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"
Jul 17 14:27:22.202226 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.202199    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"]
Jul 17 14:27:22.339541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.339513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011b3732-d6a4-440b-93c6-88a0ed9bfaf7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s\" (UID: \"011b3732-d6a4-440b-93c6-88a0ed9bfaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"
Jul 17 14:27:22.440714 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.440652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011b3732-d6a4-440b-93c6-88a0ed9bfaf7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s\" (UID: \"011b3732-d6a4-440b-93c6-88a0ed9bfaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"
Jul 17 14:27:22.441154 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.441138    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011b3732-d6a4-440b-93c6-88a0ed9bfaf7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s\" (UID: \"011b3732-d6a4-440b-93c6-88a0ed9bfaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"
Jul 17 14:27:22.505925 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.505903    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"
Jul 17 14:27:22.634054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:22.634028    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"]
Jul 17 14:27:22.636415 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:27:22.636385    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011b3732_d6a4_440b_93c6_88a0ed9bfaf7.slice/crio-ccedc43ceb023b56430b7ea8fe9f81fe76f6116b7bcd892bfbcf0cb249719e9c WatchSource:0}: Error finding container ccedc43ceb023b56430b7ea8fe9f81fe76f6116b7bcd892bfbcf0cb249719e9c: Status 404 returned error can't find the container with id ccedc43ceb023b56430b7ea8fe9f81fe76f6116b7bcd892bfbcf0cb249719e9c
Jul 17 14:27:23.592538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:23.592471    2575 generic.go:356] "Generic (PLEG): container finished" podID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerID="0d304cac7012e48aedd25de3aa703a61711235baea3167b1fde2a9712d372417" exitCode=6
Jul 17 14:27:23.592538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:23.592517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" event={"ID":"011b3732-d6a4-440b-93c6-88a0ed9bfaf7","Type":"ContainerDied","Data":"0d304cac7012e48aedd25de3aa703a61711235baea3167b1fde2a9712d372417"}
Jul 17 14:27:23.592882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:23.592557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" event={"ID":"011b3732-d6a4-440b-93c6-88a0ed9bfaf7","Type":"ContainerStarted","Data":"ccedc43ceb023b56430b7ea8fe9f81fe76f6116b7bcd892bfbcf0cb249719e9c"}
Jul 17 14:27:23.592882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:23.592704    2575 scope.go:117] "RemoveContainer" containerID="0d304cac7012e48aedd25de3aa703a61711235baea3167b1fde2a9712d372417"
Jul 17 14:27:24.598815 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:24.598787    2575 generic.go:356] "Generic (PLEG): container finished" podID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerID="aaeff90591ce6188ecb30ff1b99e6216de6e5d4683a364b20756557589283cb1" exitCode=6
Jul 17 14:27:24.599240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:24.598836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" event={"ID":"011b3732-d6a4-440b-93c6-88a0ed9bfaf7","Type":"ContainerDied","Data":"aaeff90591ce6188ecb30ff1b99e6216de6e5d4683a364b20756557589283cb1"}
Jul 17 14:27:24.599240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:24.598892    2575 scope.go:117] "RemoveContainer" containerID="0d304cac7012e48aedd25de3aa703a61711235baea3167b1fde2a9712d372417"
Jul 17 14:27:24.599240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:24.599014    2575 scope.go:117] "RemoveContainer" containerID="aaeff90591ce6188ecb30ff1b99e6216de6e5d4683a364b20756557589283cb1"
Jul 17 14:27:24.599240 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:24.599201    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s_odh-ai-gateway-infra(011b3732-d6a4-440b-93c6-88a0ed9bfaf7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7"
Jul 17 14:27:25.069150 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:25.069123    2575 scope.go:117] "RemoveContainer" containerID="9c55404efc0976ca0311581cf5f5f721571d618b500358569a75a33e0931754d"
Jul 17 14:27:25.604560 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:25.604530    2575 scope.go:117] "RemoveContainer" containerID="aaeff90591ce6188ecb30ff1b99e6216de6e5d4683a364b20756557589283cb1"
Jul 17 14:27:25.604931 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:25.604735    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s_odh-ai-gateway-infra(011b3732-d6a4-440b-93c6-88a0ed9bfaf7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7"
Jul 17 14:27:25.606005 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:25.605983    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" event={"ID":"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd","Type":"ContainerStarted","Data":"41e8bb11dd6669a13ee149f781628ddddbcca2b549945a9ca89c903558476bdc"}
Jul 17 14:27:26.612023 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:26.611985    2575 generic.go:356] "Generic (PLEG): container finished" podID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerID="41e8bb11dd6669a13ee149f781628ddddbcca2b549945a9ca89c903558476bdc" exitCode=7
Jul 17 14:27:26.612417 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:26.612061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" event={"ID":"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd","Type":"ContainerDied","Data":"41e8bb11dd6669a13ee149f781628ddddbcca2b549945a9ca89c903558476bdc"}
Jul 17 14:27:26.612417 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:26.612119    2575 scope.go:117] "RemoveContainer" containerID="9c55404efc0976ca0311581cf5f5f721571d618b500358569a75a33e0931754d"
Jul 17 14:27:26.612417 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:26.612366    2575 scope.go:117] "RemoveContainer" containerID="41e8bb11dd6669a13ee149f781628ddddbcca2b549945a9ca89c903558476bdc"
Jul 17 14:27:26.612569 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:26.612553    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d_odh-ai-gateway-infra(b9f7a78d-107a-428e-bfa4-c1f06b6ecebd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd"
Jul 17 14:27:26.664722 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:26.664696    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"]
Jul 17 14:27:27.069738 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:27.069719    2575 scope.go:117] "RemoveContainer" containerID="9f7ac78f8b1cd53954b86479639b556ac09860a5a2469a73ccbf345ddfa2a84d"
Jul 17 14:27:27.620074 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:27.620046    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" event={"ID":"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c","Type":"ContainerStarted","Data":"56a67f8493110b1f12eaa2e60292fd189ddd9e0bd288402e8bb9e3a750da33b7"}
Jul 17 14:27:27.748361 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:27.748335    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"
Jul 17 14:27:27.786886 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:27.786856    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7a78d-107a-428e-bfa4-c1f06b6ecebd-maas-api-service-ca\") pod \"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd\" (UID: \"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd\") "
Jul 17 14:27:27.787254 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:27.787232    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f7a78d-107a-428e-bfa4-c1f06b6ecebd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" (UID: "b9f7a78d-107a-428e-bfa4-c1f06b6ecebd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:27:27.888427 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:27.888364    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f7a78d-107a-428e-bfa4-c1f06b6ecebd-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:27:28.626820 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.626793    2575 generic.go:356] "Generic (PLEG): container finished" podID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerID="56a67f8493110b1f12eaa2e60292fd189ddd9e0bd288402e8bb9e3a750da33b7" exitCode=6
Jul 17 14:27:28.627195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.626871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" event={"ID":"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c","Type":"ContainerDied","Data":"56a67f8493110b1f12eaa2e60292fd189ddd9e0bd288402e8bb9e3a750da33b7"}
Jul 17 14:27:28.627195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.626929    2575 scope.go:117] "RemoveContainer" containerID="9f7ac78f8b1cd53954b86479639b556ac09860a5a2469a73ccbf345ddfa2a84d"
Jul 17 14:27:28.627195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.627161    2575 scope.go:117] "RemoveContainer" containerID="56a67f8493110b1f12eaa2e60292fd189ddd9e0bd288402e8bb9e3a750da33b7"
Jul 17 14:27:28.627430 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:28.627404    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-gxbjq_odh-ai-gateway-infra(92d9bd34-0ad3-4e70-b528-c8d8f47fe42c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c"
Jul 17 14:27:28.631161 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.631142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d" event={"ID":"b9f7a78d-107a-428e-bfa4-c1f06b6ecebd","Type":"ContainerDied","Data":"17f486b9dab8484a1bfb3fdff0369deb329d2313079e2bce30240a3836631530"}
Jul 17 14:27:28.631252 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.631165    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"
Jul 17 14:27:28.640419 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.640404    2575 scope.go:117] "RemoveContainer" containerID="41e8bb11dd6669a13ee149f781628ddddbcca2b549945a9ca89c903558476bdc"
Jul 17 14:27:28.660252 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.660229    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"]
Jul 17 14:27:28.666067 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.666046    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-ssb6d"]
Jul 17 14:27:28.670113 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:28.670090    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"]
Jul 17 14:27:29.760640 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:29.760618    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"
Jul 17 14:27:29.801370 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:29.801347    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d9bd34-0ad3-4e70-b528-c8d8f47fe42c-maas-api-service-ca\") pod \"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c\" (UID: \"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c\") "
Jul 17 14:27:29.801670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:29.801649    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d9bd34-0ad3-4e70-b528-c8d8f47fe42c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" (UID: "92d9bd34-0ad3-4e70-b528-c8d8f47fe42c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:27:29.902779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:29.902714    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d9bd34-0ad3-4e70-b528-c8d8f47fe42c-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:27:30.068903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:30.068875    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:27:30.069109 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:30.069088    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:27:30.074846 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:30.074821    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" path="/var/lib/kubelet/pods/b9f7a78d-107a-428e-bfa4-c1f06b6ecebd/volumes"
Jul 17 14:27:30.643515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:30.643486    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"
Jul 17 14:27:30.643515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:30.643498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq" event={"ID":"92d9bd34-0ad3-4e70-b528-c8d8f47fe42c","Type":"ContainerDied","Data":"75956740fd4addff9d13c32006643a7a8d0bf0b2f4e939098c89eb812d777243"}
Jul 17 14:27:30.643728 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:30.643552    2575 scope.go:117] "RemoveContainer" containerID="56a67f8493110b1f12eaa2e60292fd189ddd9e0bd288402e8bb9e3a750da33b7"
Jul 17 14:27:30.664097 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:30.664074    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"]
Jul 17 14:27:30.671957 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:30.671939    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-gxbjq"]
Jul 17 14:27:32.076329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:32.076298    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" path="/var/lib/kubelet/pods/92d9bd34-0ad3-4e70-b528-c8d8f47fe42c/volumes"
Jul 17 14:27:38.069809 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:38.069776    2575 scope.go:117] "RemoveContainer" containerID="aaeff90591ce6188ecb30ff1b99e6216de6e5d4683a364b20756557589283cb1"
Jul 17 14:27:38.683316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:38.683290    2575 generic.go:356] "Generic (PLEG): container finished" podID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerID="e97f0607660c49a2ee17cb829e0b4af61749f4e4a1201c4a1c31aba23d37955b" exitCode=6
Jul 17 14:27:38.683316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:38.683309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" event={"ID":"011b3732-d6a4-440b-93c6-88a0ed9bfaf7","Type":"ContainerDied","Data":"e97f0607660c49a2ee17cb829e0b4af61749f4e4a1201c4a1c31aba23d37955b"}
Jul 17 14:27:38.683448 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:38.683366    2575 scope.go:117] "RemoveContainer" containerID="aaeff90591ce6188ecb30ff1b99e6216de6e5d4683a364b20756557589283cb1"
Jul 17 14:27:38.683616 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:38.683600    2575 scope.go:117] "RemoveContainer" containerID="e97f0607660c49a2ee17cb829e0b4af61749f4e4a1201c4a1c31aba23d37955b"
Jul 17 14:27:38.683819 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:38.683788    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s_odh-ai-gateway-infra(011b3732-d6a4-440b-93c6-88a0ed9bfaf7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7"
Jul 17 14:27:39.711329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:39.711294    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"]
Jul 17 14:27:39.840087 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:39.840062    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"
Jul 17 14:27:39.882844 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:39.882813    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011b3732-d6a4-440b-93c6-88a0ed9bfaf7-maas-api-service-ca\") pod \"011b3732-d6a4-440b-93c6-88a0ed9bfaf7\" (UID: \"011b3732-d6a4-440b-93c6-88a0ed9bfaf7\") "
Jul 17 14:27:39.883236 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:39.883207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011b3732-d6a4-440b-93c6-88a0ed9bfaf7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "011b3732-d6a4-440b-93c6-88a0ed9bfaf7" (UID: "011b3732-d6a4-440b-93c6-88a0ed9bfaf7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:27:39.984132 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:39.984103    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011b3732-d6a4-440b-93c6-88a0ed9bfaf7-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:27:40.694917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:40.694887    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s" event={"ID":"011b3732-d6a4-440b-93c6-88a0ed9bfaf7","Type":"ContainerDied","Data":"ccedc43ceb023b56430b7ea8fe9f81fe76f6116b7bcd892bfbcf0cb249719e9c"}
Jul 17 14:27:40.694917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:40.694916    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"
Jul 17 14:27:40.695107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:40.694949    2575 scope.go:117] "RemoveContainer" containerID="e97f0607660c49a2ee17cb829e0b4af61749f4e4a1201c4a1c31aba23d37955b"
Jul 17 14:27:40.712890 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:40.712868    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"]
Jul 17 14:27:40.715880 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:40.715859    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-8rl8s"]
Jul 17 14:27:42.071723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:42.071699    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:27:42.072082 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:42.071876    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:27:42.074554 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:42.074536    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" path="/var/lib/kubelet/pods/011b3732-d6a4-440b-93c6-88a0ed9bfaf7/volumes"
Jul 17 14:27:55.068845 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:27:55.068813    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:27:55.069418 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:27:55.069046    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:28:01.542261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542226    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"]
Jul 17 14:28:01.542621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542598    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:01.542621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542609    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:01.542621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542617    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:01.542621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542623    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542634    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542639    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542649    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542655    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542661    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542666    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542683    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542689    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542743    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542750    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:01.542755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542756    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:01.543075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542762    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:01.543075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542769    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:01.543075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542777    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:01.543075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.542782    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:01.545655 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.545639    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"
Jul 17 14:28:01.548075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.548054    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:28:01.554308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.554261    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"]
Jul 17 14:28:01.655215 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.655185    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-j65qd\" (UID: \"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"
Jul 17 14:28:01.756509 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.756478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-j65qd\" (UID: \"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"
Jul 17 14:28:01.757054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.757035    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-j65qd\" (UID: \"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"
Jul 17 14:28:01.856553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.856494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"
Jul 17 14:28:01.986188 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:01.986160    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"]
Jul 17 14:28:01.988175 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:28:01.988147    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8a5031_238d_4d3a_a9cd_94cf4bc5c6e0.slice/crio-3bfeaf90bf84a8bc75070d8393bfb9cc85b58126d8da5ef8170065c79c591c5c WatchSource:0}: Error finding container 3bfeaf90bf84a8bc75070d8393bfb9cc85b58126d8da5ef8170065c79c591c5c: Status 404 returned error can't find the container with id 3bfeaf90bf84a8bc75070d8393bfb9cc85b58126d8da5ef8170065c79c591c5c
Jul 17 14:28:02.801870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:02.801830    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerID="2180937d97c6eb523f9da9db2f609334f0ee67fc20ea6bff84df420e82c7d9a5" exitCode=6
Jul 17 14:28:02.802194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:02.801910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" event={"ID":"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0","Type":"ContainerDied","Data":"2180937d97c6eb523f9da9db2f609334f0ee67fc20ea6bff84df420e82c7d9a5"}
Jul 17 14:28:02.802194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:02.801984    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" event={"ID":"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0","Type":"ContainerStarted","Data":"3bfeaf90bf84a8bc75070d8393bfb9cc85b58126d8da5ef8170065c79c591c5c"}
Jul 17 14:28:02.802194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:02.802019    2575 scope.go:117] "RemoveContainer" containerID="2180937d97c6eb523f9da9db2f609334f0ee67fc20ea6bff84df420e82c7d9a5"
Jul 17 14:28:03.809049 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:03.809018    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerID="48fc10613399cee187ba67a200c83b0c91a1ccc663b5492f6ddd8f91038a6de4" exitCode=6
Jul 17 14:28:03.809431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:03.809074    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" event={"ID":"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0","Type":"ContainerDied","Data":"48fc10613399cee187ba67a200c83b0c91a1ccc663b5492f6ddd8f91038a6de4"}
Jul 17 14:28:03.809431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:03.809136    2575 scope.go:117] "RemoveContainer" containerID="2180937d97c6eb523f9da9db2f609334f0ee67fc20ea6bff84df420e82c7d9a5"
Jul 17 14:28:03.809431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:03.809324    2575 scope.go:117] "RemoveContainer" containerID="48fc10613399cee187ba67a200c83b0c91a1ccc663b5492f6ddd8f91038a6de4"
Jul 17 14:28:03.809562 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:03.809492    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-j65qd_odh-ai-gateway-infra(3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0"
Jul 17 14:28:04.815607 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:04.815584    2575 scope.go:117] "RemoveContainer" containerID="48fc10613399cee187ba67a200c83b0c91a1ccc663b5492f6ddd8f91038a6de4"
Jul 17 14:28:04.815987 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:04.815767    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-j65qd_odh-ai-gateway-infra(3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0"
Jul 17 14:28:08.069837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:08.069804    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:28:08.070320 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:08.069977    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:28:17.069690 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:17.069662    2575 scope.go:117] "RemoveContainer" containerID="48fc10613399cee187ba67a200c83b0c91a1ccc663b5492f6ddd8f91038a6de4"
Jul 17 14:28:17.878867 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:17.878840    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerID="4ce5dda04a54c9ac4756a63464effcf137b4e16e1c4636088c30fdf9866c4159" exitCode=6
Jul 17 14:28:17.878982 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:17.878914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" event={"ID":"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0","Type":"ContainerDied","Data":"4ce5dda04a54c9ac4756a63464effcf137b4e16e1c4636088c30fdf9866c4159"}
Jul 17 14:28:17.878982 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:17.878978    2575 scope.go:117] "RemoveContainer" containerID="48fc10613399cee187ba67a200c83b0c91a1ccc663b5492f6ddd8f91038a6de4"
Jul 17 14:28:17.879326 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:17.879307    2575 scope.go:117] "RemoveContainer" containerID="4ce5dda04a54c9ac4756a63464effcf137b4e16e1c4636088c30fdf9866c4159"
Jul 17 14:28:17.879540 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:17.879521    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-j65qd_odh-ai-gateway-infra(3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0"
Jul 17 14:28:18.920208 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:18.920177    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"]
Jul 17 14:28:19.058268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.058242    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"
Jul 17 14:28:19.069396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.069374    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:28:19.189890 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.189820    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0-maas-api-service-ca\") pod \"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0\" (UID: \"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0\") "
Jul 17 14:28:19.190180 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.190156    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" (UID: "3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:28:19.290821 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.290795    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:28:19.891631 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.891548    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerStarted","Data":"0174ee1b6af2fdf9f62122c02c4bc4c51c4bab80afef614eb7b5f6e2078cb207"}
Jul 17 14:28:19.891857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.891820    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:28:19.892934 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.892912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd" event={"ID":"3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0","Type":"ContainerDied","Data":"3bfeaf90bf84a8bc75070d8393bfb9cc85b58126d8da5ef8170065c79c591c5c"}
Jul 17 14:28:19.893030 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.892936    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"
Jul 17 14:28:19.893030 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.892959    2575 scope.go:117] "RemoveContainer" containerID="4ce5dda04a54c9ac4756a63464effcf137b4e16e1c4636088c30fdf9866c4159"
Jul 17 14:28:19.966815 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.966770    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"]
Jul 17 14:28:19.968412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:19.968395    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-j65qd"]
Jul 17 14:28:20.075062 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:20.075037    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" path="/var/lib/kubelet/pods/3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0/volumes"
Jul 17 14:28:20.900685 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:20.900647    2575 generic.go:356] "Generic (PLEG): container finished" podID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerID="0174ee1b6af2fdf9f62122c02c4bc4c51c4bab80afef614eb7b5f6e2078cb207" exitCode=1
Jul 17 14:28:20.900847 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:20.900731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerDied","Data":"0174ee1b6af2fdf9f62122c02c4bc4c51c4bab80afef614eb7b5f6e2078cb207"}
Jul 17 14:28:20.900847 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:20.900786    2575 scope.go:117] "RemoveContainer" containerID="e56c5555464626edc3a019f577d5abb1d268719ce8cec0200f63145c0530da88"
Jul 17 14:28:20.900934 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:20.900910    2575 scope.go:117] "RemoveContainer" containerID="0174ee1b6af2fdf9f62122c02c4bc4c51c4bab80afef614eb7b5f6e2078cb207"
Jul 17 14:28:20.901115 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:20.901096    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:28:21.906877 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:21.906851    2575 scope.go:117] "RemoveContainer" containerID="0174ee1b6af2fdf9f62122c02c4bc4c51c4bab80afef614eb7b5f6e2078cb207"
Jul 17 14:28:21.907233 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:21.907008    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f_odh-ai-gateway-infra(534a979a-fa8e-4b5e-b3d5-4d2835152f86)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86"
Jul 17 14:28:24.208387 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.208354    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"]
Jul 17 14:28:24.354355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.354328    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:28:24.531503 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.531478    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls\") pod \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") "
Jul 17 14:28:24.531678 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.531537    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a979a-fa8e-4b5e-b3d5-4d2835152f86-openshift-service-ca\") pod \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") "
Jul 17 14:28:24.531678 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.531587    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t57qc\" (UniqueName: \"kubernetes.io/projected/534a979a-fa8e-4b5e-b3d5-4d2835152f86-kube-api-access-t57qc\") pod \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\" (UID: \"534a979a-fa8e-4b5e-b3d5-4d2835152f86\") "
Jul 17 14:28:24.531947 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.531921    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534a979a-fa8e-4b5e-b3d5-4d2835152f86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "534a979a-fa8e-4b5e-b3d5-4d2835152f86" (UID: "534a979a-fa8e-4b5e-b3d5-4d2835152f86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:28:24.533553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.533528    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534a979a-fa8e-4b5e-b3d5-4d2835152f86-kube-api-access-t57qc" (OuterVolumeSpecName: "kube-api-access-t57qc") pod "534a979a-fa8e-4b5e-b3d5-4d2835152f86" (UID: "534a979a-fa8e-4b5e-b3d5-4d2835152f86"). InnerVolumeSpecName "kube-api-access-t57qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:28:24.533659 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.533600    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "534a979a-fa8e-4b5e-b3d5-4d2835152f86" (UID: "534a979a-fa8e-4b5e-b3d5-4d2835152f86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:28:24.632699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.632668    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t57qc\" (UniqueName: \"kubernetes.io/projected/534a979a-fa8e-4b5e-b3d5-4d2835152f86-kube-api-access-t57qc\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:28:24.632699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.632698    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/534a979a-fa8e-4b5e-b3d5-4d2835152f86-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:28:24.632855 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.632712    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a979a-fa8e-4b5e-b3d5-4d2835152f86-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:28:24.920790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.920711    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"
Jul 17 14:28:24.920790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.920759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f" event={"ID":"534a979a-fa8e-4b5e-b3d5-4d2835152f86","Type":"ContainerDied","Data":"471ed54e144a41fa25ac28a5e66efb34890e2bf14e6ab80fc1baf7944d8448c8"}
Jul 17 14:28:24.920951 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.920843    2575 scope.go:117] "RemoveContainer" containerID="0174ee1b6af2fdf9f62122c02c4bc4c51c4bab80afef614eb7b5f6e2078cb207"
Jul 17 14:28:24.945129 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.945105    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"]
Jul 17 14:28:24.950132 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:24.950112    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73fe1bea-88686c55b-w5c6f"]
Jul 17 14:28:26.074586 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:26.074553    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" path="/var/lib/kubelet/pods/534a979a-fa8e-4b5e-b3d5-4d2835152f86/volumes"
Jul 17 14:28:33.174424 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174388    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"]
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174756    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174766    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f7a78d-107a-428e-bfa4-c1f06b6ecebd" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174777    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174782    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174788    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174794    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174800    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174804    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174813    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174818    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174830    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174837    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174845    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174850    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174863    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174868    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174874    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:33.174882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174879    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174934    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174943    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174949    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174954    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174960    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="011b3732-d6a4-440b-93c6-88a0ed9bfaf7" containerName="revoke-keys"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174965    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8a5031-238d-4d3a-a9cd-94cf4bc5c6e0" containerName="revoke-keys"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174970    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174975    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d9bd34-0ad3-4e70-b528-c8d8f47fe42c" containerName="revoke-keys"
Jul 17 14:28:33.175425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.174980    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:33.178056 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.178039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"
Jul 17 14:28:33.189811 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.189789    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:28:33.202672 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.202641    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"]
Jul 17 14:28:33.298779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.298743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf6a49f-ada4-4520-a5c9-a816e053df9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g\" (UID: \"6cf6a49f-ada4-4520-a5c9-a816e053df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"
Jul 17 14:28:33.400239 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.400201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf6a49f-ada4-4520-a5c9-a816e053df9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g\" (UID: \"6cf6a49f-ada4-4520-a5c9-a816e053df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"
Jul 17 14:28:33.400828 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.400809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf6a49f-ada4-4520-a5c9-a816e053df9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g\" (UID: \"6cf6a49f-ada4-4520-a5c9-a816e053df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"
Jul 17 14:28:33.487268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.487243    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"
Jul 17 14:28:33.632010 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.631855    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"]
Jul 17 14:28:33.634654 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:28:33.634622    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf6a49f_ada4_4520_a5c9_a816e053df9c.slice/crio-7ca332d479592b1524f479ad7089e79ccc7d833bd062d9d1e06b299dd28ff2ae WatchSource:0}: Error finding container 7ca332d479592b1524f479ad7089e79ccc7d833bd062d9d1e06b299dd28ff2ae: Status 404 returned error can't find the container with id 7ca332d479592b1524f479ad7089e79ccc7d833bd062d9d1e06b299dd28ff2ae
Jul 17 14:28:33.965155 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.965122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" event={"ID":"6cf6a49f-ada4-4520-a5c9-a816e053df9c","Type":"ContainerStarted","Data":"26e03d738e9b94b4cccc165f70ac529f6fcda3eb98f3fa63f06270826489320b"}
Jul 17 14:28:33.965260 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.965180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" event={"ID":"6cf6a49f-ada4-4520-a5c9-a816e053df9c","Type":"ContainerStarted","Data":"7ca332d479592b1524f479ad7089e79ccc7d833bd062d9d1e06b299dd28ff2ae"}
Jul 17 14:28:33.988761 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:33.988723    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" podStartSLOduration=0.988709443 podStartE2EDuration="988.709443ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:28:33.987135275 +0000 UTC m=+2592.374482715" watchObservedRunningTime="2026-07-17 14:28:33.988709443 +0000 UTC m=+2592.376056883"
Jul 17 14:28:34.971270 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:34.971233    2575 generic.go:356] "Generic (PLEG): container finished" podID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerID="26e03d738e9b94b4cccc165f70ac529f6fcda3eb98f3fa63f06270826489320b" exitCode=6
Jul 17 14:28:34.971671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:34.971311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" event={"ID":"6cf6a49f-ada4-4520-a5c9-a816e053df9c","Type":"ContainerDied","Data":"26e03d738e9b94b4cccc165f70ac529f6fcda3eb98f3fa63f06270826489320b"}
Jul 17 14:28:34.971671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:34.971514    2575 scope.go:117] "RemoveContainer" containerID="26e03d738e9b94b4cccc165f70ac529f6fcda3eb98f3fa63f06270826489320b"
Jul 17 14:28:35.978424 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:35.978392    2575 generic.go:356] "Generic (PLEG): container finished" podID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerID="8d1a9b495edf64ad1fcbc26e4c176c344e6c8cfc205e51b07115f2bf02a7f834" exitCode=6
Jul 17 14:28:35.978887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:35.978451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" event={"ID":"6cf6a49f-ada4-4520-a5c9-a816e053df9c","Type":"ContainerDied","Data":"8d1a9b495edf64ad1fcbc26e4c176c344e6c8cfc205e51b07115f2bf02a7f834"}
Jul 17 14:28:35.978887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:35.978495    2575 scope.go:117] "RemoveContainer" containerID="26e03d738e9b94b4cccc165f70ac529f6fcda3eb98f3fa63f06270826489320b"
Jul 17 14:28:35.978887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:35.978716    2575 scope.go:117] "RemoveContainer" containerID="8d1a9b495edf64ad1fcbc26e4c176c344e6c8cfc205e51b07115f2bf02a7f834"
Jul 17 14:28:35.979047 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:35.978934    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g_odh-ai-gateway-infra(6cf6a49f-ada4-4520-a5c9-a816e053df9c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c"
Jul 17 14:28:36.985101 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:36.985078    2575 scope.go:117] "RemoveContainer" containerID="8d1a9b495edf64ad1fcbc26e4c176c344e6c8cfc205e51b07115f2bf02a7f834"
Jul 17 14:28:36.985494 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:36.985259    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g_odh-ai-gateway-infra(6cf6a49f-ada4-4520-a5c9-a816e053df9c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c"
Jul 17 14:28:48.973935 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.973847    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"]
Jul 17 14:28:48.974432 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.974413    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:48.974510 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.974434    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:48.974510 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.974446    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:48.974510 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.974454    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:48.974510 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.974467    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:48.974510 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.974475    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:48.974738 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.974598    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:48.978837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.978818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"
Jul 17 14:28:48.988679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:48.988658    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"]
Jul 17 14:28:49.021177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:49.021151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524b8b3-6b78-4678-8256-6f26ee7d0b83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z\" (UID: \"1524b8b3-6b78-4678-8256-6f26ee7d0b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"
Jul 17 14:28:49.122237 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:49.122208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524b8b3-6b78-4678-8256-6f26ee7d0b83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z\" (UID: \"1524b8b3-6b78-4678-8256-6f26ee7d0b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"
Jul 17 14:28:49.122880 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:49.122861    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524b8b3-6b78-4678-8256-6f26ee7d0b83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z\" (UID: \"1524b8b3-6b78-4678-8256-6f26ee7d0b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"
Jul 17 14:28:49.290336 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:49.290238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"
Jul 17 14:28:49.418986 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:49.418963    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"]
Jul 17 14:28:49.421459 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:28:49.421433    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1524b8b3_6b78_4678_8256_6f26ee7d0b83.slice/crio-0d4a851aaf0f722ac292c59820718aa38f91e359d238627cd0c3f572f12db671 WatchSource:0}: Error finding container 0d4a851aaf0f722ac292c59820718aa38f91e359d238627cd0c3f572f12db671: Status 404 returned error can't find the container with id 0d4a851aaf0f722ac292c59820718aa38f91e359d238627cd0c3f572f12db671
Jul 17 14:28:50.042997 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:50.042964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" event={"ID":"1524b8b3-6b78-4678-8256-6f26ee7d0b83","Type":"ContainerStarted","Data":"35073043748d7d560a14b692bc92d7a9023cb77c138e563fd747ee846255b6c7"}
Jul 17 14:28:50.043379 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:50.043015    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" event={"ID":"1524b8b3-6b78-4678-8256-6f26ee7d0b83","Type":"ContainerStarted","Data":"0d4a851aaf0f722ac292c59820718aa38f91e359d238627cd0c3f572f12db671"}
Jul 17 14:28:50.061596 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:50.061544    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" podStartSLOduration=2.061525012 podStartE2EDuration="2.061525012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:28:50.05965118 +0000 UTC m=+2608.446998621" watchObservedRunningTime="2026-07-17 14:28:50.061525012 +0000 UTC m=+2608.448872484"
Jul 17 14:28:51.055020 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.054987    2575 generic.go:356] "Generic (PLEG): container finished" podID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerID="35073043748d7d560a14b692bc92d7a9023cb77c138e563fd747ee846255b6c7" exitCode=6
Jul 17 14:28:51.055020 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.055023    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" event={"ID":"1524b8b3-6b78-4678-8256-6f26ee7d0b83","Type":"ContainerDied","Data":"35073043748d7d560a14b692bc92d7a9023cb77c138e563fd747ee846255b6c7"}
Jul 17 14:28:51.055437 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.055203    2575 scope.go:117] "RemoveContainer" containerID="35073043748d7d560a14b692bc92d7a9023cb77c138e563fd747ee846255b6c7"
Jul 17 14:28:51.069491 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.069467    2575 scope.go:117] "RemoveContainer" containerID="8d1a9b495edf64ad1fcbc26e4c176c344e6c8cfc205e51b07115f2bf02a7f834"
Jul 17 14:28:51.594427 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.594400    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"]
Jul 17 14:28:51.594837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.594825    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="534a979a-fa8e-4b5e-b3d5-4d2835152f86" containerName="maas-api"
Jul 17 14:28:51.597830 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.597809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"
Jul 17 14:28:51.605759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.605739    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"]
Jul 17 14:28:51.642430 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.642374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dceb98d-2611-486f-8680-db010cbff66d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-rcmlk\" (UID: \"0dceb98d-2611-486f-8680-db010cbff66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"
Jul 17 14:28:51.743186 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.743165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dceb98d-2611-486f-8680-db010cbff66d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-rcmlk\" (UID: \"0dceb98d-2611-486f-8680-db010cbff66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"
Jul 17 14:28:51.743706 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.743689    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dceb98d-2611-486f-8680-db010cbff66d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-rcmlk\" (UID: \"0dceb98d-2611-486f-8680-db010cbff66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"
Jul 17 14:28:51.909176 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:51.909090    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"
Jul 17 14:28:52.037296 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.037260    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"]
Jul 17 14:28:52.039109 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:28:52.039084    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dceb98d_2611_486f_8680_db010cbff66d.slice/crio-9f28339085260f4214d2f2dbc161c794af862dd1c3849239acbbdc13c5eec366 WatchSource:0}: Error finding container 9f28339085260f4214d2f2dbc161c794af862dd1c3849239acbbdc13c5eec366: Status 404 returned error can't find the container with id 9f28339085260f4214d2f2dbc161c794af862dd1c3849239acbbdc13c5eec366
Jul 17 14:28:52.061944 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.061918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" event={"ID":"0dceb98d-2611-486f-8680-db010cbff66d","Type":"ContainerStarted","Data":"9f28339085260f4214d2f2dbc161c794af862dd1c3849239acbbdc13c5eec366"}
Jul 17 14:28:52.063614 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.063593    2575 generic.go:356] "Generic (PLEG): container finished" podID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerID="038f83f28c0ebc6a73496e99107f1e73f5bb5f5c6a2f6204a01d69d891a6d17e" exitCode=6
Jul 17 14:28:52.063720 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.063667    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" event={"ID":"6cf6a49f-ada4-4520-a5c9-a816e053df9c","Type":"ContainerDied","Data":"038f83f28c0ebc6a73496e99107f1e73f5bb5f5c6a2f6204a01d69d891a6d17e"}
Jul 17 14:28:52.063781 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.063724    2575 scope.go:117] "RemoveContainer" containerID="8d1a9b495edf64ad1fcbc26e4c176c344e6c8cfc205e51b07115f2bf02a7f834"
Jul 17 14:28:52.064003 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.063985    2575 scope.go:117] "RemoveContainer" containerID="038f83f28c0ebc6a73496e99107f1e73f5bb5f5c6a2f6204a01d69d891a6d17e"
Jul 17 14:28:52.064179 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:52.064160    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g_odh-ai-gateway-infra(6cf6a49f-ada4-4520-a5c9-a816e053df9c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c"
Jul 17 14:28:52.065541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.065522    2575 generic.go:356] "Generic (PLEG): container finished" podID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerID="ed7312a44d3a08d299d2a44a7e16596d2508cb00c904638cd71fc5ab72947d62" exitCode=6
Jul 17 14:28:52.065637 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.065586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" event={"ID":"1524b8b3-6b78-4678-8256-6f26ee7d0b83","Type":"ContainerDied","Data":"ed7312a44d3a08d299d2a44a7e16596d2508cb00c904638cd71fc5ab72947d62"}
Jul 17 14:28:52.065775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.065761    2575 scope.go:117] "RemoveContainer" containerID="ed7312a44d3a08d299d2a44a7e16596d2508cb00c904638cd71fc5ab72947d62"
Jul 17 14:28:52.065940 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:52.065923    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z_odh-ai-gateway-infra(1524b8b3-6b78-4678-8256-6f26ee7d0b83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83"
Jul 17 14:28:52.079199 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:52.079174    2575 scope.go:117] "RemoveContainer" containerID="35073043748d7d560a14b692bc92d7a9023cb77c138e563fd747ee846255b6c7"
Jul 17 14:28:53.071928 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.071896    2575 generic.go:356] "Generic (PLEG): container finished" podID="0dceb98d-2611-486f-8680-db010cbff66d" containerID="64a845cfee43abb9aa4e228022c8e948b03372c680a122778a58bf7575063144" exitCode=6
Jul 17 14:28:53.072372 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.071981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" event={"ID":"0dceb98d-2611-486f-8680-db010cbff66d","Type":"ContainerDied","Data":"64a845cfee43abb9aa4e228022c8e948b03372c680a122778a58bf7575063144"}
Jul 17 14:28:53.072372 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.072161    2575 scope.go:117] "RemoveContainer" containerID="64a845cfee43abb9aa4e228022c8e948b03372c680a122778a58bf7575063144"
Jul 17 14:28:53.075078 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.075061    2575 scope.go:117] "RemoveContainer" containerID="ed7312a44d3a08d299d2a44a7e16596d2508cb00c904638cd71fc5ab72947d62"
Jul 17 14:28:53.075239 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:53.075214    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z_odh-ai-gateway-infra(1524b8b3-6b78-4678-8256-6f26ee7d0b83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83"
Jul 17 14:28:53.097948 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.097920    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"]
Jul 17 14:28:53.240301 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.240260    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"
Jul 17 14:28:53.357106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.357035    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf6a49f-ada4-4520-a5c9-a816e053df9c-maas-api-service-ca\") pod \"6cf6a49f-ada4-4520-a5c9-a816e053df9c\" (UID: \"6cf6a49f-ada4-4520-a5c9-a816e053df9c\") "
Jul 17 14:28:53.357448 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.357429    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf6a49f-ada4-4520-a5c9-a816e053df9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6cf6a49f-ada4-4520-a5c9-a816e053df9c" (UID: "6cf6a49f-ada4-4520-a5c9-a816e053df9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:28:53.457785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:53.457752    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf6a49f-ada4-4520-a5c9-a816e053df9c-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:28:54.080247 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.080213    2575 generic.go:356] "Generic (PLEG): container finished" podID="0dceb98d-2611-486f-8680-db010cbff66d" containerID="577d8b33124e98446193bdc3363b7946886712db651c4abdee983afdc19851bc" exitCode=6
Jul 17 14:28:54.080703 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.080302    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" event={"ID":"0dceb98d-2611-486f-8680-db010cbff66d","Type":"ContainerDied","Data":"577d8b33124e98446193bdc3363b7946886712db651c4abdee983afdc19851bc"}
Jul 17 14:28:54.080703 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.080355    2575 scope.go:117] "RemoveContainer" containerID="64a845cfee43abb9aa4e228022c8e948b03372c680a122778a58bf7575063144"
Jul 17 14:28:54.080703 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.080523    2575 scope.go:117] "RemoveContainer" containerID="577d8b33124e98446193bdc3363b7946886712db651c4abdee983afdc19851bc"
Jul 17 14:28:54.080881 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:54.080722    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-rcmlk_odh-ai-gateway-infra(0dceb98d-2611-486f-8680-db010cbff66d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" podUID="0dceb98d-2611-486f-8680-db010cbff66d"
Jul 17 14:28:54.081601 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.081583    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"
Jul 17 14:28:54.081676 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.081611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g" event={"ID":"6cf6a49f-ada4-4520-a5c9-a816e053df9c","Type":"ContainerDied","Data":"7ca332d479592b1524f479ad7089e79ccc7d833bd062d9d1e06b299dd28ff2ae"}
Jul 17 14:28:54.091031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.091012    2575 scope.go:117] "RemoveContainer" containerID="038f83f28c0ebc6a73496e99107f1e73f5bb5f5c6a2f6204a01d69d891a6d17e"
Jul 17 14:28:54.111584 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.111563    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"]
Jul 17 14:28:54.118779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:54.118758    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-h2p6g"]
Jul 17 14:28:55.088686 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:55.088642    2575 scope.go:117] "RemoveContainer" containerID="577d8b33124e98446193bdc3363b7946886712db651c4abdee983afdc19851bc"
Jul 17 14:28:55.089075 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:28:55.088826    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-rcmlk_odh-ai-gateway-infra(0dceb98d-2611-486f-8680-db010cbff66d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" podUID="0dceb98d-2611-486f-8680-db010cbff66d"
Jul 17 14:28:56.076235 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:28:56.076199    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" path="/var/lib/kubelet/pods/6cf6a49f-ada4-4520-a5c9-a816e053df9c/volumes"
Jul 17 14:29:05.069159 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:05.069131    2575 scope.go:117] "RemoveContainer" containerID="ed7312a44d3a08d299d2a44a7e16596d2508cb00c904638cd71fc5ab72947d62"
Jul 17 14:29:06.138182 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:06.138110    2575 generic.go:356] "Generic (PLEG): container finished" podID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerID="8832f136b66a742c030f63a97163ca65268b8ac46537b9de007f32f0dfecc9d1" exitCode=6
Jul 17 14:29:06.138557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:06.138180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" event={"ID":"1524b8b3-6b78-4678-8256-6f26ee7d0b83","Type":"ContainerDied","Data":"8832f136b66a742c030f63a97163ca65268b8ac46537b9de007f32f0dfecc9d1"}
Jul 17 14:29:06.138557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:06.138232    2575 scope.go:117] "RemoveContainer" containerID="ed7312a44d3a08d299d2a44a7e16596d2508cb00c904638cd71fc5ab72947d62"
Jul 17 14:29:06.138557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:06.138456    2575 scope.go:117] "RemoveContainer" containerID="8832f136b66a742c030f63a97163ca65268b8ac46537b9de007f32f0dfecc9d1"
Jul 17 14:29:06.138691 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:06.138670    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z_odh-ai-gateway-infra(1524b8b3-6b78-4678-8256-6f26ee7d0b83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83"
Jul 17 14:29:07.165034 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:07.165003    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"]
Jul 17 14:29:07.295539 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:07.295517    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"
Jul 17 14:29:07.370183 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:07.370150    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524b8b3-6b78-4678-8256-6f26ee7d0b83-maas-api-service-ca\") pod \"1524b8b3-6b78-4678-8256-6f26ee7d0b83\" (UID: \"1524b8b3-6b78-4678-8256-6f26ee7d0b83\") "
Jul 17 14:29:07.370522 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:07.370498    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1524b8b3-6b78-4678-8256-6f26ee7d0b83-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1524b8b3-6b78-4678-8256-6f26ee7d0b83" (UID: "1524b8b3-6b78-4678-8256-6f26ee7d0b83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:29:07.471441 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:07.471423    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524b8b3-6b78-4678-8256-6f26ee7d0b83-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:29:08.149673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:08.149644    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"
Jul 17 14:29:08.149834 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:08.149678    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z" event={"ID":"1524b8b3-6b78-4678-8256-6f26ee7d0b83","Type":"ContainerDied","Data":"0d4a851aaf0f722ac292c59820718aa38f91e359d238627cd0c3f572f12db671"}
Jul 17 14:29:08.149834 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:08.149744    2575 scope.go:117] "RemoveContainer" containerID="8832f136b66a742c030f63a97163ca65268b8ac46537b9de007f32f0dfecc9d1"
Jul 17 14:29:08.166735 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:08.166713    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"]
Jul 17 14:29:08.171417 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:08.171395    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2hp8z"]
Jul 17 14:29:10.069031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:10.069003    2575 scope.go:117] "RemoveContainer" containerID="577d8b33124e98446193bdc3363b7946886712db651c4abdee983afdc19851bc"
Jul 17 14:29:10.074906 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:10.074883    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" path="/var/lib/kubelet/pods/1524b8b3-6b78-4678-8256-6f26ee7d0b83/volumes"
Jul 17 14:29:11.166396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:11.166319    2575 generic.go:356] "Generic (PLEG): container finished" podID="0dceb98d-2611-486f-8680-db010cbff66d" containerID="50456212de7012c04afd68937e3f9de9b2b9d67c319574e591038d6daa30a3b6" exitCode=6
Jul 17 14:29:11.166741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:11.166396    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" event={"ID":"0dceb98d-2611-486f-8680-db010cbff66d","Type":"ContainerDied","Data":"50456212de7012c04afd68937e3f9de9b2b9d67c319574e591038d6daa30a3b6"}
Jul 17 14:29:11.166741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:11.166455    2575 scope.go:117] "RemoveContainer" containerID="577d8b33124e98446193bdc3363b7946886712db651c4abdee983afdc19851bc"
Jul 17 14:29:11.166741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:11.166692    2575 scope.go:117] "RemoveContainer" containerID="50456212de7012c04afd68937e3f9de9b2b9d67c319574e591038d6daa30a3b6"
Jul 17 14:29:11.166885 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:11.166864    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-rcmlk_odh-ai-gateway-infra(0dceb98d-2611-486f-8680-db010cbff66d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" podUID="0dceb98d-2611-486f-8680-db010cbff66d"
Jul 17 14:29:12.207190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.200643    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"]
Jul 17 14:29:12.308946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.308909    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"]
Jul 17 14:29:12.309362 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309349    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309367    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309380    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.309412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309385    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.309412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309395    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309400    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309410    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.309412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309415    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309427    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309436    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309453    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309460    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309568    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309580    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309590    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309602    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf6a49f-ada4-4520-a5c9-a816e053df9c" containerName="revoke-keys"
Jul 17 14:29:12.309767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.309610    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:12.314121 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.314103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"
Jul 17 14:29:12.321661 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.321640    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"]
Jul 17 14:29:12.331827 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.331793    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"
Jul 17 14:29:12.411109 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.411081    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dceb98d-2611-486f-8680-db010cbff66d-maas-api-service-ca\") pod \"0dceb98d-2611-486f-8680-db010cbff66d\" (UID: \"0dceb98d-2611-486f-8680-db010cbff66d\") "
Jul 17 14:29:12.411223 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.411178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46975d86-d8b7-468a-a628-ec7550e0427d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx\" (UID: \"46975d86-d8b7-468a-a628-ec7550e0427d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"
Jul 17 14:29:12.411461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.411442    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dceb98d-2611-486f-8680-db010cbff66d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dceb98d-2611-486f-8680-db010cbff66d" (UID: "0dceb98d-2611-486f-8680-db010cbff66d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:29:12.512345 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.512317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46975d86-d8b7-468a-a628-ec7550e0427d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx\" (UID: \"46975d86-d8b7-468a-a628-ec7550e0427d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"
Jul 17 14:29:12.512475 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.512433    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dceb98d-2611-486f-8680-db010cbff66d-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:29:12.512897 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.512878    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46975d86-d8b7-468a-a628-ec7550e0427d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx\" (UID: \"46975d86-d8b7-468a-a628-ec7550e0427d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"
Jul 17 14:29:12.637940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.637920    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"
Jul 17 14:29:12.981197 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:12.981170    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"]
Jul 17 14:29:12.983712 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:29:12.983668    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46975d86_d8b7_468a_a628_ec7550e0427d.slice/crio-7a4a9dd21d3ca30dc77e3dde46c30afff564a30353de0913a53ec8a0befd8c3d WatchSource:0}: Error finding container 7a4a9dd21d3ca30dc77e3dde46c30afff564a30353de0913a53ec8a0befd8c3d: Status 404 returned error can't find the container with id 7a4a9dd21d3ca30dc77e3dde46c30afff564a30353de0913a53ec8a0befd8c3d
Jul 17 14:29:13.177976 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:13.177951    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"
Jul 17 14:29:13.178125 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:13.177978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk" event={"ID":"0dceb98d-2611-486f-8680-db010cbff66d","Type":"ContainerDied","Data":"9f28339085260f4214d2f2dbc161c794af862dd1c3849239acbbdc13c5eec366"}
Jul 17 14:29:13.178125 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:13.178039    2575 scope.go:117] "RemoveContainer" containerID="50456212de7012c04afd68937e3f9de9b2b9d67c319574e591038d6daa30a3b6"
Jul 17 14:29:13.179430 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:13.179402    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" event={"ID":"46975d86-d8b7-468a-a628-ec7550e0427d","Type":"ContainerStarted","Data":"7a4a9dd21d3ca30dc77e3dde46c30afff564a30353de0913a53ec8a0befd8c3d"}
Jul 17 14:29:13.206730 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:13.206706    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"]
Jul 17 14:29:13.216666 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:13.216635    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rcmlk"]
Jul 17 14:29:14.074919 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:14.074886    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dceb98d-2611-486f-8680-db010cbff66d" path="/var/lib/kubelet/pods/0dceb98d-2611-486f-8680-db010cbff66d/volumes"
Jul 17 14:29:14.186104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:14.186062    2575 generic.go:356] "Generic (PLEG): container finished" podID="46975d86-d8b7-468a-a628-ec7550e0427d" containerID="f165af5339d3b8a4b23be2a58f65dd6429d14c0dd36ed95750e157e03d01723a" exitCode=6
Jul 17 14:29:14.186266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:14.186108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" event={"ID":"46975d86-d8b7-468a-a628-ec7550e0427d","Type":"ContainerDied","Data":"f165af5339d3b8a4b23be2a58f65dd6429d14c0dd36ed95750e157e03d01723a"}
Jul 17 14:29:14.186351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:14.186336    2575 scope.go:117] "RemoveContainer" containerID="f165af5339d3b8a4b23be2a58f65dd6429d14c0dd36ed95750e157e03d01723a"
Jul 17 14:29:15.192619 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:15.192537    2575 generic.go:356] "Generic (PLEG): container finished" podID="46975d86-d8b7-468a-a628-ec7550e0427d" containerID="ee38fa43e4914a588f8594a90c6608cb3a6dd61b2443992896311301a84ab7af" exitCode=6
Jul 17 14:29:15.192619 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:15.192614    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" event={"ID":"46975d86-d8b7-468a-a628-ec7550e0427d","Type":"ContainerDied","Data":"ee38fa43e4914a588f8594a90c6608cb3a6dd61b2443992896311301a84ab7af"}
Jul 17 14:29:15.193054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:15.192673    2575 scope.go:117] "RemoveContainer" containerID="f165af5339d3b8a4b23be2a58f65dd6429d14c0dd36ed95750e157e03d01723a"
Jul 17 14:29:15.193054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:15.192851    2575 scope.go:117] "RemoveContainer" containerID="ee38fa43e4914a588f8594a90c6608cb3a6dd61b2443992896311301a84ab7af"
Jul 17 14:29:15.193054 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:15.193031    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx_odh-ai-gateway-infra(46975d86-d8b7-468a-a628-ec7550e0427d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" podUID="46975d86-d8b7-468a-a628-ec7550e0427d"
Jul 17 14:29:16.198993 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:16.198966    2575 scope.go:117] "RemoveContainer" containerID="ee38fa43e4914a588f8594a90c6608cb3a6dd61b2443992896311301a84ab7af"
Jul 17 14:29:16.199377 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:16.199140    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx_odh-ai-gateway-infra(46975d86-d8b7-468a-a628-ec7550e0427d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" podUID="46975d86-d8b7-468a-a628-ec7550e0427d"
Jul 17 14:29:29.069725 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:29.069688    2575 scope.go:117] "RemoveContainer" containerID="ee38fa43e4914a588f8594a90c6608cb3a6dd61b2443992896311301a84ab7af"
Jul 17 14:29:30.264141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:30.264110    2575 generic.go:356] "Generic (PLEG): container finished" podID="46975d86-d8b7-468a-a628-ec7550e0427d" containerID="e44df1b0da35a1b823cbabec7f4f0d10af5cc8271b67eb81875391139acf3db0" exitCode=6
Jul 17 14:29:30.264542 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:30.264179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" event={"ID":"46975d86-d8b7-468a-a628-ec7550e0427d","Type":"ContainerDied","Data":"e44df1b0da35a1b823cbabec7f4f0d10af5cc8271b67eb81875391139acf3db0"}
Jul 17 14:29:30.264542 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:30.264234    2575 scope.go:117] "RemoveContainer" containerID="ee38fa43e4914a588f8594a90c6608cb3a6dd61b2443992896311301a84ab7af"
Jul 17 14:29:30.264542 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:30.264473    2575 scope.go:117] "RemoveContainer" containerID="e44df1b0da35a1b823cbabec7f4f0d10af5cc8271b67eb81875391139acf3db0"
Jul 17 14:29:30.264721 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:30.264662    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx_odh-ai-gateway-infra(46975d86-d8b7-468a-a628-ec7550e0427d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" podUID="46975d86-d8b7-468a-a628-ec7550e0427d"
Jul 17 14:29:31.294353 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:31.294322    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"]
Jul 17 14:29:31.442580 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:31.442555    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"
Jul 17 14:29:31.572336 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:31.572235    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46975d86-d8b7-468a-a628-ec7550e0427d-maas-api-service-ca\") pod \"46975d86-d8b7-468a-a628-ec7550e0427d\" (UID: \"46975d86-d8b7-468a-a628-ec7550e0427d\") "
Jul 17 14:29:31.572614 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:31.572595    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46975d86-d8b7-468a-a628-ec7550e0427d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46975d86-d8b7-468a-a628-ec7550e0427d" (UID: "46975d86-d8b7-468a-a628-ec7550e0427d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:29:31.673126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:31.673100    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46975d86-d8b7-468a-a628-ec7550e0427d-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:29:32.277244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:32.277212    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx" event={"ID":"46975d86-d8b7-468a-a628-ec7550e0427d","Type":"ContainerDied","Data":"7a4a9dd21d3ca30dc77e3dde46c30afff564a30353de0913a53ec8a0befd8c3d"}
Jul 17 14:29:32.277244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:32.277237    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"
Jul 17 14:29:32.277467 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:32.277268    2575 scope.go:117] "RemoveContainer" containerID="e44df1b0da35a1b823cbabec7f4f0d10af5cc8271b67eb81875391139acf3db0"
Jul 17 14:29:32.294535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:32.294514    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"]
Jul 17 14:29:32.301199 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:32.301177    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-f72mx"]
Jul 17 14:29:33.209722 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.209692    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"]
Jul 17 14:29:33.210045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210033    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:33.210100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:33.210100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210057    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:33.210100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210062    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:33.210100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210074    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210080    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210092    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210097    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210107    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210112    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210168    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210176    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1524b8b3-6b78-4678-8256-6f26ee7d0b83" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210182    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210187    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210193    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210199    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" containerName="revoke-keys"
Jul 17 14:29:33.210331 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.210204    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:33.214536 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.214520    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"
Jul 17 14:29:33.218756 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.218738    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:29:33.225238 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.225212    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"]
Jul 17 14:29:33.387612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.387581    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh\" (UID: \"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"
Jul 17 14:29:33.488408 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.488378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh\" (UID: \"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"
Jul 17 14:29:33.488993 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.488972    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh\" (UID: \"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"
Jul 17 14:29:33.524929 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.524908    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"
Jul 17 14:29:33.857104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:33.857071    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"]
Jul 17 14:29:33.859640 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:29:33.859607    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6ed4b8_8811_4bcd_a5f7_da21154c8c6c.slice/crio-86e24180d2368f7ee931e1bf32518a05a23b34ef824578a6c70d91c8603d5265 WatchSource:0}: Error finding container 86e24180d2368f7ee931e1bf32518a05a23b34ef824578a6c70d91c8603d5265: Status 404 returned error can't find the container with id 86e24180d2368f7ee931e1bf32518a05a23b34ef824578a6c70d91c8603d5265
Jul 17 14:29:34.075412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:34.075380    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46975d86-d8b7-468a-a628-ec7550e0427d" path="/var/lib/kubelet/pods/46975d86-d8b7-468a-a628-ec7550e0427d/volumes"
Jul 17 14:29:34.289841 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:34.289805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" event={"ID":"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c","Type":"ContainerStarted","Data":"a2b013b09a3fbc9f98f168f5bcbf8ecf247d2cd3b99110d67d7648e1c79f2fdd"}
Jul 17 14:29:34.290020 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:34.289867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" event={"ID":"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c","Type":"ContainerStarted","Data":"86e24180d2368f7ee931e1bf32518a05a23b34ef824578a6c70d91c8603d5265"}
Jul 17 14:29:34.308241 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:34.308192    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" podStartSLOduration=1.3081773270000001 podStartE2EDuration="1.308177327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:29:34.306790996 +0000 UTC m=+2652.694138436" watchObservedRunningTime="2026-07-17 14:29:34.308177327 +0000 UTC m=+2652.695524766"
Jul 17 14:29:35.296201 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:35.296161    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerID="a2b013b09a3fbc9f98f168f5bcbf8ecf247d2cd3b99110d67d7648e1c79f2fdd" exitCode=6
Jul 17 14:29:35.296730 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:35.296246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" event={"ID":"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c","Type":"ContainerDied","Data":"a2b013b09a3fbc9f98f168f5bcbf8ecf247d2cd3b99110d67d7648e1c79f2fdd"}
Jul 17 14:29:35.296730 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:35.296480    2575 scope.go:117] "RemoveContainer" containerID="a2b013b09a3fbc9f98f168f5bcbf8ecf247d2cd3b99110d67d7648e1c79f2fdd"
Jul 17 14:29:36.301775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:36.301746    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerID="b4217027460d1ecafcb7795dad1d7e6d3f2714a332b10a3b0b5470d4dc90a453" exitCode=6
Jul 17 14:29:36.302213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:36.301828    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" event={"ID":"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c","Type":"ContainerDied","Data":"b4217027460d1ecafcb7795dad1d7e6d3f2714a332b10a3b0b5470d4dc90a453"}
Jul 17 14:29:36.302213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:36.301883    2575 scope.go:117] "RemoveContainer" containerID="a2b013b09a3fbc9f98f168f5bcbf8ecf247d2cd3b99110d67d7648e1c79f2fdd"
Jul 17 14:29:36.302213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:36.301968    2575 scope.go:117] "RemoveContainer" containerID="b4217027460d1ecafcb7795dad1d7e6d3f2714a332b10a3b0b5470d4dc90a453"
Jul 17 14:29:36.302213 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:36.302171    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh_odh-ai-gateway-infra(7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c"
Jul 17 14:29:37.307984 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:37.307955    2575 scope.go:117] "RemoveContainer" containerID="b4217027460d1ecafcb7795dad1d7e6d3f2714a332b10a3b0b5470d4dc90a453"
Jul 17 14:29:37.308401 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:37.308130    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh_odh-ai-gateway-infra(7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c"
Jul 17 14:29:39.043141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.043101    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"]
Jul 17 14:29:39.043708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.043686    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:39.043708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.043710    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dceb98d-2611-486f-8680-db010cbff66d" containerName="revoke-keys"
Jul 17 14:29:39.046886 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.046866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"
Jul 17 14:29:39.059993 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.059967    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"]
Jul 17 14:29:39.130072 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.130042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ef8e27-664e-4999-9364-d6a342c7fa85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk\" (UID: \"e9ef8e27-664e-4999-9364-d6a342c7fa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"
Jul 17 14:29:39.230922 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.230889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ef8e27-664e-4999-9364-d6a342c7fa85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk\" (UID: \"e9ef8e27-664e-4999-9364-d6a342c7fa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"
Jul 17 14:29:39.231524 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.231506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ef8e27-664e-4999-9364-d6a342c7fa85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk\" (UID: \"e9ef8e27-664e-4999-9364-d6a342c7fa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"
Jul 17 14:29:39.356264 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.356186    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"
Jul 17 14:29:39.488099 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:39.488075    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"]
Jul 17 14:29:39.490661 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:29:39.490629    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ef8e27_664e_4999_9364_d6a342c7fa85.slice/crio-23ca97020cdc111582753c16cf99dbb9c599404d8834ac2286d04f502c841d1d WatchSource:0}: Error finding container 23ca97020cdc111582753c16cf99dbb9c599404d8834ac2286d04f502c841d1d: Status 404 returned error can't find the container with id 23ca97020cdc111582753c16cf99dbb9c599404d8834ac2286d04f502c841d1d
Jul 17 14:29:40.322900 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:40.322811    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerID="135b512fbb1b514570155ded496334306ac0ce7f84e19c5f3e1d4a2ac9a3f168" exitCode=6
Jul 17 14:29:40.323216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:40.322905    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" event={"ID":"e9ef8e27-664e-4999-9364-d6a342c7fa85","Type":"ContainerDied","Data":"135b512fbb1b514570155ded496334306ac0ce7f84e19c5f3e1d4a2ac9a3f168"}
Jul 17 14:29:40.323216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:40.322949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" event={"ID":"e9ef8e27-664e-4999-9364-d6a342c7fa85","Type":"ContainerStarted","Data":"23ca97020cdc111582753c16cf99dbb9c599404d8834ac2286d04f502c841d1d"}
Jul 17 14:29:40.323216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:40.323036    2575 scope.go:117] "RemoveContainer" containerID="135b512fbb1b514570155ded496334306ac0ce7f84e19c5f3e1d4a2ac9a3f168"
Jul 17 14:29:41.328673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:41.328630    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerID="38dede5fd7961ebb85177a300fb3f349dbaf7a4f8de572bab004bece12e450ef" exitCode=6
Jul 17 14:29:41.329084 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:41.328718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" event={"ID":"e9ef8e27-664e-4999-9364-d6a342c7fa85","Type":"ContainerDied","Data":"38dede5fd7961ebb85177a300fb3f349dbaf7a4f8de572bab004bece12e450ef"}
Jul 17 14:29:41.329084 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:41.328801    2575 scope.go:117] "RemoveContainer" containerID="135b512fbb1b514570155ded496334306ac0ce7f84e19c5f3e1d4a2ac9a3f168"
Jul 17 14:29:41.329084 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:41.328956    2575 scope.go:117] "RemoveContainer" containerID="38dede5fd7961ebb85177a300fb3f349dbaf7a4f8de572bab004bece12e450ef"
Jul 17 14:29:41.329205 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:41.329132    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk_odh-ai-gateway-infra(e9ef8e27-664e-4999-9364-d6a342c7fa85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85"
Jul 17 14:29:42.335545 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:42.335518    2575 scope.go:117] "RemoveContainer" containerID="38dede5fd7961ebb85177a300fb3f349dbaf7a4f8de572bab004bece12e450ef"
Jul 17 14:29:42.335894 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:42.335688    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk_odh-ai-gateway-infra(e9ef8e27-664e-4999-9364-d6a342c7fa85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85"
Jul 17 14:29:51.069604 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.069572    2575 scope.go:117] "RemoveContainer" containerID="b4217027460d1ecafcb7795dad1d7e6d3f2714a332b10a3b0b5470d4dc90a453"
Jul 17 14:29:51.645575 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.645501    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"]
Jul 17 14:29:51.648946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.648928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"
Jul 17 14:29:51.658327 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.658299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"]
Jul 17 14:29:51.730141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.730113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23190dd3-a60f-4d52-ae26-0712d0bccd7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-6z9qc\" (UID: \"23190dd3-a60f-4d52-ae26-0712d0bccd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"
Jul 17 14:29:51.831347 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.831320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23190dd3-a60f-4d52-ae26-0712d0bccd7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-6z9qc\" (UID: \"23190dd3-a60f-4d52-ae26-0712d0bccd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"
Jul 17 14:29:51.831852 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.831836    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23190dd3-a60f-4d52-ae26-0712d0bccd7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-6z9qc\" (UID: \"23190dd3-a60f-4d52-ae26-0712d0bccd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"
Jul 17 14:29:51.959208 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:51.959157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"
Jul 17 14:29:52.107694 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.107666    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"]
Jul 17 14:29:52.108831 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:29:52.108809    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23190dd3_a60f_4d52_ae26_0712d0bccd7e.slice/crio-0beb20be6a73a20d52cd3b8fe73e9a27a4b9ec77feb152de387587ad4c1bbdf8 WatchSource:0}: Error finding container 0beb20be6a73a20d52cd3b8fe73e9a27a4b9ec77feb152de387587ad4c1bbdf8: Status 404 returned error can't find the container with id 0beb20be6a73a20d52cd3b8fe73e9a27a4b9ec77feb152de387587ad4c1bbdf8
Jul 17 14:29:52.384571 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.384542    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerID="b4a5f68c35e7420b08080c6a7b6deec063242e19f2ca15d5c787e19e5bf00e15" exitCode=6
Jul 17 14:29:52.384723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.384618    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" event={"ID":"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c","Type":"ContainerDied","Data":"b4a5f68c35e7420b08080c6a7b6deec063242e19f2ca15d5c787e19e5bf00e15"}
Jul 17 14:29:52.384723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.384687    2575 scope.go:117] "RemoveContainer" containerID="b4217027460d1ecafcb7795dad1d7e6d3f2714a332b10a3b0b5470d4dc90a453"
Jul 17 14:29:52.384963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.384936    2575 scope.go:117] "RemoveContainer" containerID="b4a5f68c35e7420b08080c6a7b6deec063242e19f2ca15d5c787e19e5bf00e15"
Jul 17 14:29:52.385187 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:52.385167    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh_odh-ai-gateway-infra(7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c"
Jul 17 14:29:52.386251 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.386225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" event={"ID":"23190dd3-a60f-4d52-ae26-0712d0bccd7e","Type":"ContainerStarted","Data":"ab996fe381a737e2aed9ce650f87675a0156cadce166c15d0b794fb224c1040c"}
Jul 17 14:29:52.386358 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.386264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" event={"ID":"23190dd3-a60f-4d52-ae26-0712d0bccd7e","Type":"ContainerStarted","Data":"0beb20be6a73a20d52cd3b8fe73e9a27a4b9ec77feb152de387587ad4c1bbdf8"}
Jul 17 14:29:52.424290 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:52.419532    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" podStartSLOduration=1.419514872 podStartE2EDuration="1.419514872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:29:52.418679862 +0000 UTC m=+2670.806027302" watchObservedRunningTime="2026-07-17 14:29:52.419514872 +0000 UTC m=+2670.806862311"
Jul 17 14:29:53.394333 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.394269    2575 generic.go:356] "Generic (PLEG): container finished" podID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerID="ab996fe381a737e2aed9ce650f87675a0156cadce166c15d0b794fb224c1040c" exitCode=6
Jul 17 14:29:53.394333 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.394311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" event={"ID":"23190dd3-a60f-4d52-ae26-0712d0bccd7e","Type":"ContainerDied","Data":"ab996fe381a737e2aed9ce650f87675a0156cadce166c15d0b794fb224c1040c"}
Jul 17 14:29:53.394750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.394528    2575 scope.go:117] "RemoveContainer" containerID="ab996fe381a737e2aed9ce650f87675a0156cadce166c15d0b794fb224c1040c"
Jul 17 14:29:53.412420 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.412395    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"]
Jul 17 14:29:53.541119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.541098    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"
Jul 17 14:29:53.647966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.647905    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c-maas-api-service-ca\") pod \"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c\" (UID: \"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c\") "
Jul 17 14:29:53.648226 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.648206    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" (UID: "7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:29:53.748844 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:53.748811    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:29:54.400674 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.400632    2575 generic.go:356] "Generic (PLEG): container finished" podID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerID="fd2a0ac032806c4ca01e1c2d24285d1622d460f4195a195437f7e132aeb0307d" exitCode=6
Jul 17 14:29:54.401146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.400723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" event={"ID":"23190dd3-a60f-4d52-ae26-0712d0bccd7e","Type":"ContainerDied","Data":"fd2a0ac032806c4ca01e1c2d24285d1622d460f4195a195437f7e132aeb0307d"}
Jul 17 14:29:54.401146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.400791    2575 scope.go:117] "RemoveContainer" containerID="ab996fe381a737e2aed9ce650f87675a0156cadce166c15d0b794fb224c1040c"
Jul 17 14:29:54.401146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.400896    2575 scope.go:117] "RemoveContainer" containerID="fd2a0ac032806c4ca01e1c2d24285d1622d460f4195a195437f7e132aeb0307d"
Jul 17 14:29:54.401340 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:54.401155    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-6z9qc_odh-ai-gateway-infra(23190dd3-a60f-4d52-ae26-0712d0bccd7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e"
Jul 17 14:29:54.402085 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.402067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh" event={"ID":"7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c","Type":"ContainerDied","Data":"86e24180d2368f7ee931e1bf32518a05a23b34ef824578a6c70d91c8603d5265"}
Jul 17 14:29:54.402140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.402086    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"
Jul 17 14:29:54.411497 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.411488    2575 scope.go:117] "RemoveContainer" containerID="b4a5f68c35e7420b08080c6a7b6deec063242e19f2ca15d5c787e19e5bf00e15"
Jul 17 14:29:54.432634 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.432611    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"]
Jul 17 14:29:54.441938 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:54.441918    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-d7lbh"]
Jul 17 14:29:55.409642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:55.409610    2575 scope.go:117] "RemoveContainer" containerID="fd2a0ac032806c4ca01e1c2d24285d1622d460f4195a195437f7e132aeb0307d"
Jul 17 14:29:55.410014 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:55.409769    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-6z9qc_odh-ai-gateway-infra(23190dd3-a60f-4d52-ae26-0712d0bccd7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e"
Jul 17 14:29:56.069299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:56.069259    2575 scope.go:117] "RemoveContainer" containerID="38dede5fd7961ebb85177a300fb3f349dbaf7a4f8de572bab004bece12e450ef"
Jul 17 14:29:56.074728 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:56.074697    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" path="/var/lib/kubelet/pods/7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c/volumes"
Jul 17 14:29:56.416516 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:56.416432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" event={"ID":"e9ef8e27-664e-4999-9364-d6a342c7fa85","Type":"ContainerStarted","Data":"eaec4c67b587d964abc2e5528a336792608f84f741935f01c1bddb37eec7b11a"}
Jul 17 14:29:56.436595 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:56.436543    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" podStartSLOduration=17.436526661 podStartE2EDuration="17.436526661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:29:56.436140279 +0000 UTC m=+2674.823487734" watchObservedRunningTime="2026-07-17 14:29:56.436526661 +0000 UTC m=+2674.823874101"
Jul 17 14:29:57.422821 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:57.422788    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerID="eaec4c67b587d964abc2e5528a336792608f84f741935f01c1bddb37eec7b11a" exitCode=6
Jul 17 14:29:57.423328 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:57.422862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" event={"ID":"e9ef8e27-664e-4999-9364-d6a342c7fa85","Type":"ContainerDied","Data":"eaec4c67b587d964abc2e5528a336792608f84f741935f01c1bddb37eec7b11a"}
Jul 17 14:29:57.423328 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:57.422921    2575 scope.go:117] "RemoveContainer" containerID="38dede5fd7961ebb85177a300fb3f349dbaf7a4f8de572bab004bece12e450ef"
Jul 17 14:29:57.423328 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:57.423186    2575 scope.go:117] "RemoveContainer" containerID="eaec4c67b587d964abc2e5528a336792608f84f741935f01c1bddb37eec7b11a"
Jul 17 14:29:57.423465 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:29:57.423399    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk_odh-ai-gateway-infra(e9ef8e27-664e-4999-9364-d6a342c7fa85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85"
Jul 17 14:29:57.456027 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:57.455998    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"]
Jul 17 14:29:58.566109 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:58.566087    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"
Jul 17 14:29:58.698223 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:58.698146    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ef8e27-664e-4999-9364-d6a342c7fa85-maas-api-service-ca\") pod \"e9ef8e27-664e-4999-9364-d6a342c7fa85\" (UID: \"e9ef8e27-664e-4999-9364-d6a342c7fa85\") "
Jul 17 14:29:58.698518 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:58.698497    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ef8e27-664e-4999-9364-d6a342c7fa85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9ef8e27-664e-4999-9364-d6a342c7fa85" (UID: "e9ef8e27-664e-4999-9364-d6a342c7fa85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:29:58.798968 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:58.798936    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ef8e27-664e-4999-9364-d6a342c7fa85-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:29:59.440830 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:59.440796    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"
Jul 17 14:29:59.440830 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:59.440829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk" event={"ID":"e9ef8e27-664e-4999-9364-d6a342c7fa85","Type":"ContainerDied","Data":"23ca97020cdc111582753c16cf99dbb9c599404d8834ac2286d04f502c841d1d"}
Jul 17 14:29:59.441067 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:59.440883    2575 scope.go:117] "RemoveContainer" containerID="eaec4c67b587d964abc2e5528a336792608f84f741935f01c1bddb37eec7b11a"
Jul 17 14:29:59.462524 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:59.462490    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"]
Jul 17 14:29:59.468626 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:29:59.468604    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vzzmk"]
Jul 17 14:30:00.075705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.075671    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" path="/var/lib/kubelet/pods/e9ef8e27-664e-4999-9364-d6a342c7fa85/volumes"
Jul 17 14:30:00.133670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.133644    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd"]
Jul 17 14:30:00.134008 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.133997    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:00.134055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134010    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:00.134055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134024    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.134055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134030    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.134055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134051    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134057    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134064    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134069    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134084    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134090    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134149    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134157    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134163    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:00.134172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134171    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:00.134441 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.134179    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:00.138801 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.138783    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd"
Jul 17 14:30:00.148291 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.148234    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd"]
Jul 17 14:30:00.149070 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.149045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd"
Jul 17 14:30:00.282113 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.282091    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd"]
Jul 17 14:30:00.284360 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:30:00.284335    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0206d1e4_855a_4da5_85d3_979274f00c22.slice/crio-49693e5e8eebf9b81a501b68a792efebfdd3dc61e67c965cb7ef8284f07eef9c WatchSource:0}: Error finding container 49693e5e8eebf9b81a501b68a792efebfdd3dc61e67c965cb7ef8284f07eef9c: Status 404 returned error can't find the container with id 49693e5e8eebf9b81a501b68a792efebfdd3dc61e67c965cb7ef8284f07eef9c
Jul 17 14:30:00.448787 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:00.448707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd" event={"ID":"0206d1e4-855a-4da5-85d3-979274f00c22","Type":"ContainerStarted","Data":"49693e5e8eebf9b81a501b68a792efebfdd3dc61e67c965cb7ef8284f07eef9c"}
Jul 17 14:30:01.458512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:01.458436    2575 generic.go:356] "Generic (PLEG): container finished" podID="0206d1e4-855a-4da5-85d3-979274f00c22" containerID="f26e6782469b23ace5fa68f98aad2424bf03015d610bdf9b41db63fdf1a67ee6" exitCode=0
Jul 17 14:30:01.458512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:01.458499    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd" event={"ID":"0206d1e4-855a-4da5-85d3-979274f00c22","Type":"ContainerDied","Data":"f26e6782469b23ace5fa68f98aad2424bf03015d610bdf9b41db63fdf1a67ee6"}
Jul 17 14:30:02.596515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:02.596492    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd"
Jul 17 14:30:03.469790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:03.469755    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd" event={"ID":"0206d1e4-855a-4da5-85d3-979274f00c22","Type":"ContainerDied","Data":"49693e5e8eebf9b81a501b68a792efebfdd3dc61e67c965cb7ef8284f07eef9c"}
Jul 17 14:30:03.469790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:03.469790    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738310-7t5cd"
Jul 17 14:30:03.470000 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:03.469812    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49693e5e8eebf9b81a501b68a792efebfdd3dc61e67c965cb7ef8284f07eef9c"
Jul 17 14:30:09.069682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:09.069650    2575 scope.go:117] "RemoveContainer" containerID="fd2a0ac032806c4ca01e1c2d24285d1622d460f4195a195437f7e132aeb0307d"
Jul 17 14:30:09.500074 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:09.500043    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" event={"ID":"23190dd3-a60f-4d52-ae26-0712d0bccd7e","Type":"ContainerStarted","Data":"8e822234635ae6afaf4c09fcc60c1c921ee316cf2a923b3b97006669f68f989c"}
Jul 17 14:30:10.506230 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:10.506197    2575 generic.go:356] "Generic (PLEG): container finished" podID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerID="8e822234635ae6afaf4c09fcc60c1c921ee316cf2a923b3b97006669f68f989c" exitCode=6
Jul 17 14:30:10.506701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:10.506294    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" event={"ID":"23190dd3-a60f-4d52-ae26-0712d0bccd7e","Type":"ContainerDied","Data":"8e822234635ae6afaf4c09fcc60c1c921ee316cf2a923b3b97006669f68f989c"}
Jul 17 14:30:10.506701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:10.506359    2575 scope.go:117] "RemoveContainer" containerID="fd2a0ac032806c4ca01e1c2d24285d1622d460f4195a195437f7e132aeb0307d"
Jul 17 14:30:10.506785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:10.506712    2575 scope.go:117] "RemoveContainer" containerID="8e822234635ae6afaf4c09fcc60c1c921ee316cf2a923b3b97006669f68f989c"
Jul 17 14:30:10.506960 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:10.506942    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-6z9qc_odh-ai-gateway-infra(23190dd3-a60f-4d52-ae26-0712d0bccd7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e"
Jul 17 14:30:10.536288 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:10.536244    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"]
Jul 17 14:30:11.648535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:11.648508    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"
Jul 17 14:30:11.826189 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:11.826099    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23190dd3-a60f-4d52-ae26-0712d0bccd7e-maas-api-service-ca\") pod \"23190dd3-a60f-4d52-ae26-0712d0bccd7e\" (UID: \"23190dd3-a60f-4d52-ae26-0712d0bccd7e\") "
Jul 17 14:30:11.826499 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:11.826476    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23190dd3-a60f-4d52-ae26-0712d0bccd7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23190dd3-a60f-4d52-ae26-0712d0bccd7e" (UID: "23190dd3-a60f-4d52-ae26-0712d0bccd7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:30:11.927515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:11.927474    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23190dd3-a60f-4d52-ae26-0712d0bccd7e-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:30:12.414084 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414005    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"]
Jul 17 14:30:12.414413 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414400    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414414    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414428    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414434    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414446    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:12.414462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414451    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ef8e27-664e-4999-9364-d6a342c7fa85" containerName="revoke-keys"
Jul 17 14:30:12.414462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414459    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0206d1e4-855a-4da5-85d3-979274f00c22" containerName="cleanup"
Jul 17 14:30:12.414462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414464    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0206d1e4-855a-4da5-85d3-979274f00c22" containerName="cleanup"
Jul 17 14:30:12.414665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414476    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414481    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414540    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0206d1e4-855a-4da5-85d3-979274f00c22" containerName="cleanup"
Jul 17 14:30:12.414665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414548    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414554    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:12.414665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.414562    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6ed4b8-8811-4bcd-a5f7-da21154c8c6c" containerName="revoke-keys"
Jul 17 14:30:12.417541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.417526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"
Jul 17 14:30:12.436367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.436334    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"]
Jul 17 14:30:12.519214 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.519186    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"
Jul 17 14:30:12.519447 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.519228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc" event={"ID":"23190dd3-a60f-4d52-ae26-0712d0bccd7e","Type":"ContainerDied","Data":"0beb20be6a73a20d52cd3b8fe73e9a27a4b9ec77feb152de387587ad4c1bbdf8"}
Jul 17 14:30:12.519447 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.519309    2575 scope.go:117] "RemoveContainer" containerID="8e822234635ae6afaf4c09fcc60c1c921ee316cf2a923b3b97006669f68f989c"
Jul 17 14:30:12.533261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.533234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f83e3ee-55c5-48c6-8095-5dc869a3d07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4\" (UID: \"5f83e3ee-55c5-48c6-8095-5dc869a3d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"
Jul 17 14:30:12.554757 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.554725    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"]
Jul 17 14:30:12.557762 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.557740    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-6z9qc"]
Jul 17 14:30:12.634351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.634319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f83e3ee-55c5-48c6-8095-5dc869a3d07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4\" (UID: \"5f83e3ee-55c5-48c6-8095-5dc869a3d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"
Jul 17 14:30:12.634926 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.634908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f83e3ee-55c5-48c6-8095-5dc869a3d07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4\" (UID: \"5f83e3ee-55c5-48c6-8095-5dc869a3d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"
Jul 17 14:30:12.726604 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.726574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"
Jul 17 14:30:12.862637 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:12.862610    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"]
Jul 17 14:30:12.864455 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:30:12.864431    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f83e3ee_55c5_48c6_8095_5dc869a3d07c.slice/crio-65ac707583cc5e9be6243a1e463731fd28bf15967dba701e010ea1e151b12fd2 WatchSource:0}: Error finding container 65ac707583cc5e9be6243a1e463731fd28bf15967dba701e010ea1e151b12fd2: Status 404 returned error can't find the container with id 65ac707583cc5e9be6243a1e463731fd28bf15967dba701e010ea1e151b12fd2
Jul 17 14:30:13.526065 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:13.526034    2575 generic.go:356] "Generic (PLEG): container finished" podID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerID="7ddcec6974501259cb568e362f1a578ba70f611c07547a56bf4f36f9e1cb5fcb" exitCode=6
Jul 17 14:30:13.526189 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:13.526093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" event={"ID":"5f83e3ee-55c5-48c6-8095-5dc869a3d07c","Type":"ContainerDied","Data":"7ddcec6974501259cb568e362f1a578ba70f611c07547a56bf4f36f9e1cb5fcb"}
Jul 17 14:30:13.526189 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:13.526145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" event={"ID":"5f83e3ee-55c5-48c6-8095-5dc869a3d07c","Type":"ContainerStarted","Data":"65ac707583cc5e9be6243a1e463731fd28bf15967dba701e010ea1e151b12fd2"}
Jul 17 14:30:13.526368 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:13.526357    2575 scope.go:117] "RemoveContainer" containerID="7ddcec6974501259cb568e362f1a578ba70f611c07547a56bf4f36f9e1cb5fcb"
Jul 17 14:30:14.075101 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:14.075026    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" path="/var/lib/kubelet/pods/23190dd3-a60f-4d52-ae26-0712d0bccd7e/volumes"
Jul 17 14:30:14.533202 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:14.533170    2575 generic.go:356] "Generic (PLEG): container finished" podID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerID="f70d6c79880e6570dd241ad8843e2d82afcca3a9aec0d59d4dc264a83c9b74cb" exitCode=6
Jul 17 14:30:14.533411 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:14.533227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" event={"ID":"5f83e3ee-55c5-48c6-8095-5dc869a3d07c","Type":"ContainerDied","Data":"f70d6c79880e6570dd241ad8843e2d82afcca3a9aec0d59d4dc264a83c9b74cb"}
Jul 17 14:30:14.533411 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:14.533272    2575 scope.go:117] "RemoveContainer" containerID="7ddcec6974501259cb568e362f1a578ba70f611c07547a56bf4f36f9e1cb5fcb"
Jul 17 14:30:14.533507 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:14.533444    2575 scope.go:117] "RemoveContainer" containerID="f70d6c79880e6570dd241ad8843e2d82afcca3a9aec0d59d4dc264a83c9b74cb"
Jul 17 14:30:14.533710 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:14.533687    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4_odh-ai-gateway-infra(5f83e3ee-55c5-48c6-8095-5dc869a3d07c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c"
Jul 17 14:30:15.539572 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:15.539546    2575 scope.go:117] "RemoveContainer" containerID="f70d6c79880e6570dd241ad8843e2d82afcca3a9aec0d59d4dc264a83c9b74cb"
Jul 17 14:30:15.539924 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:15.539746    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4_odh-ai-gateway-infra(5f83e3ee-55c5-48c6-8095-5dc869a3d07c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c"
Jul 17 14:30:16.092138 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.092101    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"]
Jul 17 14:30:16.092857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.092836    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23190dd3-a60f-4d52-ae26-0712d0bccd7e" containerName="revoke-keys"
Jul 17 14:30:16.097884 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.097866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.100031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.100010    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8ed37e19-serving-cert\""
Jul 17 14:30:16.111823 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.111783    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"]
Jul 17 14:30:16.165546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.165519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ae230-2d00-42ff-aedf-d58dd37facc4-openshift-service-ca\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.165703 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.165615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxqz\" (UniqueName: \"kubernetes.io/projected/d19ae230-2d00-42ff-aedf-d58dd37facc4-kube-api-access-jkxqz\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.165703 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.165647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.266241 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.266206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxqz\" (UniqueName: \"kubernetes.io/projected/d19ae230-2d00-42ff-aedf-d58dd37facc4-kube-api-access-jkxqz\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.266428 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.266254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.266428 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.266343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ae230-2d00-42ff-aedf-d58dd37facc4-openshift-service-ca\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.266526 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:16.266422    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-serving-cert: secret "maas-api-e2e-mt-8ed37e19-serving-cert" not found
Jul 17 14:30:16.266526 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:16.266494    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls podName:d19ae230-2d00-42ff-aedf-d58dd37facc4 nodeName:}" failed. No retries permitted until 2026-07-17 14:30:16.766474079 +0000 UTC m=+2695.153821510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls") pod "maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs" (UID: "d19ae230-2d00-42ff-aedf-d58dd37facc4") : secret "maas-api-e2e-mt-8ed37e19-serving-cert" not found
Jul 17 14:30:16.266975 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.266957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ae230-2d00-42ff-aedf-d58dd37facc4-openshift-service-ca\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.275250 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.275221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxqz\" (UniqueName: \"kubernetes.io/projected/d19ae230-2d00-42ff-aedf-d58dd37facc4-kube-api-access-jkxqz\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.769613 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.769580    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:16.771963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:16.771940    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls\") pod \"maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:17.007455 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:17.007414    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:17.154803 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:17.154769    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"]
Jul 17 14:30:17.157743 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:30:17.157708    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19ae230_2d00_42ff_aedf_d58dd37facc4.slice/crio-e89fc31b6a52b3b54fdea47f07241d2a429d3a0754d79b187ca8ac543dc13d80 WatchSource:0}: Error finding container e89fc31b6a52b3b54fdea47f07241d2a429d3a0754d79b187ca8ac543dc13d80: Status 404 returned error can't find the container with id e89fc31b6a52b3b54fdea47f07241d2a429d3a0754d79b187ca8ac543dc13d80
Jul 17 14:30:17.159611 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:17.159588    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 14:30:17.549954 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:17.549924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs" event={"ID":"d19ae230-2d00-42ff-aedf-d58dd37facc4","Type":"ContainerStarted","Data":"e89fc31b6a52b3b54fdea47f07241d2a429d3a0754d79b187ca8ac543dc13d80"}
Jul 17 14:30:18.556086 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:18.556049    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs" event={"ID":"d19ae230-2d00-42ff-aedf-d58dd37facc4","Type":"ContainerStarted","Data":"027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e"}
Jul 17 14:30:18.556528 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:18.556141    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:18.580944 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:18.580889    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs" podStartSLOduration=2.222993186 podStartE2EDuration="2.580874932s" totalImagesPullingTime="357.881746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:30:18.580816463 +0000 UTC m=+2696.968163912" watchObservedRunningTime="2026-07-17 14:30:18.580874932 +0000 UTC m=+2696.968222372"
Jul 17 14:30:22.195216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:22.195189    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:30:22.199056 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:22.199034    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:30:22.204182 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:22.204163    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:30:22.205538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:22.205515    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:30:22.207708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:22.207692    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:30:22.214212 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:22.214192    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:30:24.568193 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:24.568164    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:30.068999 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:30.068964    2575 scope.go:117] "RemoveContainer" containerID="f70d6c79880e6570dd241ad8843e2d82afcca3a9aec0d59d4dc264a83c9b74cb"
Jul 17 14:30:30.615904 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:30.615867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" event={"ID":"5f83e3ee-55c5-48c6-8095-5dc869a3d07c","Type":"ContainerStarted","Data":"63257e243f17e6e18851ddd5d3f4a6094fc7d9263a0da5b4d092d69d58391676"}
Jul 17 14:30:30.635763 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:30.635667    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" podStartSLOduration=18.635652476 podStartE2EDuration="18.635652476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:30:30.633754158 +0000 UTC m=+2709.021101597" watchObservedRunningTime="2026-07-17 14:30:30.635652476 +0000 UTC m=+2709.022999914"
Jul 17 14:30:31.622270 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:31.622236    2575 generic.go:356] "Generic (PLEG): container finished" podID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerID="63257e243f17e6e18851ddd5d3f4a6094fc7d9263a0da5b4d092d69d58391676" exitCode=6
Jul 17 14:30:31.622711 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:31.622307    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" event={"ID":"5f83e3ee-55c5-48c6-8095-5dc869a3d07c","Type":"ContainerDied","Data":"63257e243f17e6e18851ddd5d3f4a6094fc7d9263a0da5b4d092d69d58391676"}
Jul 17 14:30:31.622711 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:31.622366    2575 scope.go:117] "RemoveContainer" containerID="f70d6c79880e6570dd241ad8843e2d82afcca3a9aec0d59d4dc264a83c9b74cb"
Jul 17 14:30:31.622711 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:31.622602    2575 scope.go:117] "RemoveContainer" containerID="63257e243f17e6e18851ddd5d3f4a6094fc7d9263a0da5b4d092d69d58391676"
Jul 17 14:30:31.622862 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:31.622808    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4_odh-ai-gateway-infra(5f83e3ee-55c5-48c6-8095-5dc869a3d07c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c"
Jul 17 14:30:31.650838 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:31.650804    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"]
Jul 17 14:30:32.757548 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:32.757524    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"
Jul 17 14:30:32.805299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:32.805254    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f83e3ee-55c5-48c6-8095-5dc869a3d07c-maas-api-service-ca\") pod \"5f83e3ee-55c5-48c6-8095-5dc869a3d07c\" (UID: \"5f83e3ee-55c5-48c6-8095-5dc869a3d07c\") "
Jul 17 14:30:32.805597 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:32.805573    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f83e3ee-55c5-48c6-8095-5dc869a3d07c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f83e3ee-55c5-48c6-8095-5dc869a3d07c" (UID: "5f83e3ee-55c5-48c6-8095-5dc869a3d07c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:30:32.906813 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:32.906740    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f83e3ee-55c5-48c6-8095-5dc869a3d07c-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:30:33.240584 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.240543    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"]
Jul 17 14:30:33.241015 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.240996    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241017    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241029    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241037    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241050    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241058    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241358 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241185    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241358 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241197    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.241358 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.241209    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" containerName="revoke-keys"
Jul 17 14:30:33.244483 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.244463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"
Jul 17 14:30:33.254141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.254113    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"]
Jul 17 14:30:33.310073 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.310036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bface8d-5522-4723-8adb-2acfa31f9ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f\" (UID: \"2bface8d-5522-4723-8adb-2acfa31f9ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"
Jul 17 14:30:33.410779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.410738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bface8d-5522-4723-8adb-2acfa31f9ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f\" (UID: \"2bface8d-5522-4723-8adb-2acfa31f9ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"
Jul 17 14:30:33.411385 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.411363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bface8d-5522-4723-8adb-2acfa31f9ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f\" (UID: \"2bface8d-5522-4723-8adb-2acfa31f9ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"
Jul 17 14:30:33.555437 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.555346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"
Jul 17 14:30:33.634167 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.634129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4" event={"ID":"5f83e3ee-55c5-48c6-8095-5dc869a3d07c","Type":"ContainerDied","Data":"65ac707583cc5e9be6243a1e463731fd28bf15967dba701e010ea1e151b12fd2"}
Jul 17 14:30:33.634376 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.634191    2575 scope.go:117] "RemoveContainer" containerID="63257e243f17e6e18851ddd5d3f4a6094fc7d9263a0da5b4d092d69d58391676"
Jul 17 14:30:33.634376 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.634139    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"
Jul 17 14:30:33.671666 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.671636    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"]
Jul 17 14:30:33.678708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.678677    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-sllf4"]
Jul 17 14:30:33.698364 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:33.698342    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"]
Jul 17 14:30:33.700054 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:30:33.700030    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bface8d_5522_4723_8adb_2acfa31f9ed5.slice/crio-88c65365a476294145b2a003a584e46df897f3834db5a564a3abf39d74982696 WatchSource:0}: Error finding container 88c65365a476294145b2a003a584e46df897f3834db5a564a3abf39d74982696: Status 404 returned error can't find the container with id 88c65365a476294145b2a003a584e46df897f3834db5a564a3abf39d74982696
Jul 17 14:30:34.075694 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:34.075664    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f83e3ee-55c5-48c6-8095-5dc869a3d07c" path="/var/lib/kubelet/pods/5f83e3ee-55c5-48c6-8095-5dc869a3d07c/volumes"
Jul 17 14:30:34.640769 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:34.640673    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerID="3bf43e52d060103bd48ef648d315c91910175418d7ec3803d66fa8b4bb22d92c" exitCode=6
Jul 17 14:30:34.640939 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:34.640761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" event={"ID":"2bface8d-5522-4723-8adb-2acfa31f9ed5","Type":"ContainerDied","Data":"3bf43e52d060103bd48ef648d315c91910175418d7ec3803d66fa8b4bb22d92c"}
Jul 17 14:30:34.640939 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:34.640837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" event={"ID":"2bface8d-5522-4723-8adb-2acfa31f9ed5","Type":"ContainerStarted","Data":"88c65365a476294145b2a003a584e46df897f3834db5a564a3abf39d74982696"}
Jul 17 14:30:34.641066 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:34.640960    2575 scope.go:117] "RemoveContainer" containerID="3bf43e52d060103bd48ef648d315c91910175418d7ec3803d66fa8b4bb22d92c"
Jul 17 14:30:35.648013 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:35.647930    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerID="8de69b176eaaac36e5203b8a37916f510194be9884e0f2672bec245fb6fb9a7a" exitCode=6
Jul 17 14:30:35.648013 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:35.647981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" event={"ID":"2bface8d-5522-4723-8adb-2acfa31f9ed5","Type":"ContainerDied","Data":"8de69b176eaaac36e5203b8a37916f510194be9884e0f2672bec245fb6fb9a7a"}
Jul 17 14:30:35.648477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:35.648038    2575 scope.go:117] "RemoveContainer" containerID="3bf43e52d060103bd48ef648d315c91910175418d7ec3803d66fa8b4bb22d92c"
Jul 17 14:30:35.648477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:35.648198    2575 scope.go:117] "RemoveContainer" containerID="8de69b176eaaac36e5203b8a37916f510194be9884e0f2672bec245fb6fb9a7a"
Jul 17 14:30:35.648477 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:35.648454    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f_odh-ai-gateway-infra(2bface8d-5522-4723-8adb-2acfa31f9ed5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5"
Jul 17 14:30:36.654307 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:36.654267    2575 scope.go:117] "RemoveContainer" containerID="8de69b176eaaac36e5203b8a37916f510194be9884e0f2672bec245fb6fb9a7a"
Jul 17 14:30:36.654700 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:36.654463    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f_odh-ai-gateway-infra(2bface8d-5522-4723-8adb-2acfa31f9ed5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5"
Jul 17 14:30:39.245227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.244420    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"]
Jul 17 14:30:39.245227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.244671    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs" podUID="d19ae230-2d00-42ff-aedf-d58dd37facc4" containerName="maas-api" containerID="cri-o://027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e" gracePeriod=30
Jul 17 14:30:39.487195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.487173    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:39.561036 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.560972    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ae230-2d00-42ff-aedf-d58dd37facc4-openshift-service-ca\") pod \"d19ae230-2d00-42ff-aedf-d58dd37facc4\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") "
Jul 17 14:30:39.561036 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.561010    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkxqz\" (UniqueName: \"kubernetes.io/projected/d19ae230-2d00-42ff-aedf-d58dd37facc4-kube-api-access-jkxqz\") pod \"d19ae230-2d00-42ff-aedf-d58dd37facc4\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") "
Jul 17 14:30:39.561196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.561154    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls\") pod \"d19ae230-2d00-42ff-aedf-d58dd37facc4\" (UID: \"d19ae230-2d00-42ff-aedf-d58dd37facc4\") "
Jul 17 14:30:39.561353 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.561330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19ae230-2d00-42ff-aedf-d58dd37facc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d19ae230-2d00-42ff-aedf-d58dd37facc4" (UID: "d19ae230-2d00-42ff-aedf-d58dd37facc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:30:39.561457 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.561421    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ae230-2d00-42ff-aedf-d58dd37facc4-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:30:39.563002 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.562984    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d19ae230-2d00-42ff-aedf-d58dd37facc4" (UID: "d19ae230-2d00-42ff-aedf-d58dd37facc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:30:39.563099 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.563036    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19ae230-2d00-42ff-aedf-d58dd37facc4-kube-api-access-jkxqz" (OuterVolumeSpecName: "kube-api-access-jkxqz") pod "d19ae230-2d00-42ff-aedf-d58dd37facc4" (UID: "d19ae230-2d00-42ff-aedf-d58dd37facc4"). InnerVolumeSpecName "kube-api-access-jkxqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:30:39.662740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.662718    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ae230-2d00-42ff-aedf-d58dd37facc4-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:30:39.662740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.662742    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxqz\" (UniqueName: \"kubernetes.io/projected/d19ae230-2d00-42ff-aedf-d58dd37facc4-kube-api-access-jkxqz\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:30:39.670903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.670878    2575 generic.go:356] "Generic (PLEG): container finished" podID="d19ae230-2d00-42ff-aedf-d58dd37facc4" containerID="027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e" exitCode=0
Jul 17 14:30:39.670998 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.670926    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"
Jul 17 14:30:39.671040 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.670927    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs" event={"ID":"d19ae230-2d00-42ff-aedf-d58dd37facc4","Type":"ContainerDied","Data":"027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e"}
Jul 17 14:30:39.671075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.671046    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs" event={"ID":"d19ae230-2d00-42ff-aedf-d58dd37facc4","Type":"ContainerDied","Data":"e89fc31b6a52b3b54fdea47f07241d2a429d3a0754d79b187ca8ac543dc13d80"}
Jul 17 14:30:39.671125 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.671075    2575 scope.go:117] "RemoveContainer" containerID="027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e"
Jul 17 14:30:39.685929 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.685908    2575 scope.go:117] "RemoveContainer" containerID="027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e"
Jul 17 14:30:39.686201 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:39.686175    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e\": container with ID starting with 027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e not found: ID does not exist" containerID="027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e"
Jul 17 14:30:39.686260 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.686208    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e"} err="failed to get container status \"027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e\": rpc error: code = NotFound desc = could not find container \"027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e\": container with ID starting with 027f6d82636ecf62cb1d7efc7067863bd43a36fd6c4e3c391b22dcae7c01094e not found: ID does not exist"
Jul 17 14:30:39.699940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.699920    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"]
Jul 17 14:30:39.704158 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:39.704133    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8ed37e19-5d6d489568-cqfgs"]
Jul 17 14:30:40.074863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:40.074827    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19ae230-2d00-42ff-aedf-d58dd37facc4" path="/var/lib/kubelet/pods/d19ae230-2d00-42ff-aedf-d58dd37facc4/volumes"
Jul 17 14:30:50.069341 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:50.069298    2575 scope.go:117] "RemoveContainer" containerID="8de69b176eaaac36e5203b8a37916f510194be9884e0f2672bec245fb6fb9a7a"
Jul 17 14:30:50.727422 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:50.727392    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerID="a72f3b69517b06b90129a8b1dec1a49a6569b40b944e3d67f712b534468e67f0" exitCode=6
Jul 17 14:30:50.727535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:50.727465    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" event={"ID":"2bface8d-5522-4723-8adb-2acfa31f9ed5","Type":"ContainerDied","Data":"a72f3b69517b06b90129a8b1dec1a49a6569b40b944e3d67f712b534468e67f0"}
Jul 17 14:30:50.727535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:50.727523    2575 scope.go:117] "RemoveContainer" containerID="8de69b176eaaac36e5203b8a37916f510194be9884e0f2672bec245fb6fb9a7a"
Jul 17 14:30:50.727780 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:50.727764    2575 scope.go:117] "RemoveContainer" containerID="a72f3b69517b06b90129a8b1dec1a49a6569b40b944e3d67f712b534468e67f0"
Jul 17 14:30:50.727969 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:50.727953    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f_odh-ai-gateway-infra(2bface8d-5522-4723-8adb-2acfa31f9ed5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5"
Jul 17 14:30:51.708425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.708392    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"]
Jul 17 14:30:51.708962 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.708941    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19ae230-2d00-42ff-aedf-d58dd37facc4" containerName="maas-api"
Jul 17 14:30:51.709060 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.708964    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19ae230-2d00-42ff-aedf-d58dd37facc4" containerName="maas-api"
Jul 17 14:30:51.709120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.709087    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19ae230-2d00-42ff-aedf-d58dd37facc4" containerName="maas-api"
Jul 17 14:30:51.713483 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.713461    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"
Jul 17 14:30:51.721308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.721268    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"]
Jul 17 14:30:51.765590 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.765562    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"]
Jul 17 14:30:51.866018 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.865988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d245ad2-ab6f-4a70-8938-fe432207d893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-66rvw\" (UID: \"5d245ad2-ab6f-4a70-8938-fe432207d893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"
Jul 17 14:30:51.897003 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.896982    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"
Jul 17 14:30:51.967193 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.967171    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bface8d-5522-4723-8adb-2acfa31f9ed5-maas-api-service-ca\") pod \"2bface8d-5522-4723-8adb-2acfa31f9ed5\" (UID: \"2bface8d-5522-4723-8adb-2acfa31f9ed5\") "
Jul 17 14:30:51.967378 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.967363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d245ad2-ab6f-4a70-8938-fe432207d893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-66rvw\" (UID: \"5d245ad2-ab6f-4a70-8938-fe432207d893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"
Jul 17 14:30:51.967570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.967545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bface8d-5522-4723-8adb-2acfa31f9ed5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bface8d-5522-4723-8adb-2acfa31f9ed5" (UID: "2bface8d-5522-4723-8adb-2acfa31f9ed5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:30:51.967959 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:51.967942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d245ad2-ab6f-4a70-8938-fe432207d893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-66rvw\" (UID: \"5d245ad2-ab6f-4a70-8938-fe432207d893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"
Jul 17 14:30:52.024108 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.024083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"
Jul 17 14:30:52.068037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.068008    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bface8d-5522-4723-8adb-2acfa31f9ed5-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:30:52.155971 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.155943    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"]
Jul 17 14:30:52.158150 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:30:52.158120    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d245ad2_ab6f_4a70_8938_fe432207d893.slice/crio-658b837595928cc4da2922df627a2869a6b06cd3c625f75828407e5a7360fc56 WatchSource:0}: Error finding container 658b837595928cc4da2922df627a2869a6b06cd3c625f75828407e5a7360fc56: Status 404 returned error can't find the container with id 658b837595928cc4da2922df627a2869a6b06cd3c625f75828407e5a7360fc56
Jul 17 14:30:52.741525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.741497    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"
Jul 17 14:30:52.741989 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.741531    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f" event={"ID":"2bface8d-5522-4723-8adb-2acfa31f9ed5","Type":"ContainerDied","Data":"88c65365a476294145b2a003a584e46df897f3834db5a564a3abf39d74982696"}
Jul 17 14:30:52.741989 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.741591    2575 scope.go:117] "RemoveContainer" containerID="a72f3b69517b06b90129a8b1dec1a49a6569b40b944e3d67f712b534468e67f0"
Jul 17 14:30:52.743264 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.743239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" event={"ID":"5d245ad2-ab6f-4a70-8938-fe432207d893","Type":"ContainerStarted","Data":"5e15d290f4482ffad26ea24ba2bd3720c419c2b80d421bff645f339f066ec013"}
Jul 17 14:30:52.743393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.743378    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" event={"ID":"5d245ad2-ab6f-4a70-8938-fe432207d893","Type":"ContainerStarted","Data":"658b837595928cc4da2922df627a2869a6b06cd3c625f75828407e5a7360fc56"}
Jul 17 14:30:52.759194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.759171    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"]
Jul 17 14:30:52.766191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.766169    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-hvb5f"]
Jul 17 14:30:52.782177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:52.782138    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" podStartSLOduration=1.782126255 podStartE2EDuration="1.782126255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:30:52.779858734 +0000 UTC m=+2731.167206173" watchObservedRunningTime="2026-07-17 14:30:52.782126255 +0000 UTC m=+2731.169473695"
Jul 17 14:30:53.751525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:53.751494    2575 generic.go:356] "Generic (PLEG): container finished" podID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerID="5e15d290f4482ffad26ea24ba2bd3720c419c2b80d421bff645f339f066ec013" exitCode=6
Jul 17 14:30:53.751941 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:53.751560    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" event={"ID":"5d245ad2-ab6f-4a70-8938-fe432207d893","Type":"ContainerDied","Data":"5e15d290f4482ffad26ea24ba2bd3720c419c2b80d421bff645f339f066ec013"}
Jul 17 14:30:53.751941 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:53.751764    2575 scope.go:117] "RemoveContainer" containerID="5e15d290f4482ffad26ea24ba2bd3720c419c2b80d421bff645f339f066ec013"
Jul 17 14:30:54.075780 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:54.075724    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" path="/var/lib/kubelet/pods/2bface8d-5522-4723-8adb-2acfa31f9ed5/volumes"
Jul 17 14:30:54.758291 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:54.758245    2575 generic.go:356] "Generic (PLEG): container finished" podID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerID="86d54a8a8b8063619ce6bee706371a7534743f06dc56a6feca012c6953aedc78" exitCode=6
Jul 17 14:30:54.758741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:54.758338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" event={"ID":"5d245ad2-ab6f-4a70-8938-fe432207d893","Type":"ContainerDied","Data":"86d54a8a8b8063619ce6bee706371a7534743f06dc56a6feca012c6953aedc78"}
Jul 17 14:30:54.758741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:54.758449    2575 scope.go:117] "RemoveContainer" containerID="5e15d290f4482ffad26ea24ba2bd3720c419c2b80d421bff645f339f066ec013"
Jul 17 14:30:54.758741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:54.758666    2575 scope.go:117] "RemoveContainer" containerID="86d54a8a8b8063619ce6bee706371a7534743f06dc56a6feca012c6953aedc78"
Jul 17 14:30:54.758922 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:54.758901    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-66rvw_odh-ai-gateway-infra(5d245ad2-ab6f-4a70-8938-fe432207d893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893"
Jul 17 14:30:55.764978 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:30:55.764950    2575 scope.go:117] "RemoveContainer" containerID="86d54a8a8b8063619ce6bee706371a7534743f06dc56a6feca012c6953aedc78"
Jul 17 14:30:55.765374 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:30:55.765121    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-66rvw_odh-ai-gateway-infra(5d245ad2-ab6f-4a70-8938-fe432207d893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893"
Jul 17 14:31:08.068875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:08.068841    2575 scope.go:117] "RemoveContainer" containerID="86d54a8a8b8063619ce6bee706371a7534743f06dc56a6feca012c6953aedc78"
Jul 17 14:31:08.831446 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:08.831409    2575 generic.go:356] "Generic (PLEG): container finished" podID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerID="561fa10fe452e3860db69d0a258402a0304a897ce35974cbfbd0ef1e92c5fa6d" exitCode=6
Jul 17 14:31:08.831613 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:08.831484    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" event={"ID":"5d245ad2-ab6f-4a70-8938-fe432207d893","Type":"ContainerDied","Data":"561fa10fe452e3860db69d0a258402a0304a897ce35974cbfbd0ef1e92c5fa6d"}
Jul 17 14:31:08.831613 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:08.831541    2575 scope.go:117] "RemoveContainer" containerID="86d54a8a8b8063619ce6bee706371a7534743f06dc56a6feca012c6953aedc78"
Jul 17 14:31:08.831838 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:08.831819    2575 scope.go:117] "RemoveContainer" containerID="561fa10fe452e3860db69d0a258402a0304a897ce35974cbfbd0ef1e92c5fa6d"
Jul 17 14:31:08.832114 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:08.832088    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-66rvw_odh-ai-gateway-infra(5d245ad2-ab6f-4a70-8938-fe432207d893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893"
Jul 17 14:31:09.864564 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:09.864532    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"]
Jul 17 14:31:10.001493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.001455    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"
Jul 17 14:31:10.121771 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.121684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d245ad2-ab6f-4a70-8938-fe432207d893-maas-api-service-ca\") pod \"5d245ad2-ab6f-4a70-8938-fe432207d893\" (UID: \"5d245ad2-ab6f-4a70-8938-fe432207d893\") "
Jul 17 14:31:10.122042 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.122019    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d245ad2-ab6f-4a70-8938-fe432207d893-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d245ad2-ab6f-4a70-8938-fe432207d893" (UID: "5d245ad2-ab6f-4a70-8938-fe432207d893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:31:10.222632 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.222602    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d245ad2-ab6f-4a70-8938-fe432207d893-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:31:10.843898 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.843864    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw" event={"ID":"5d245ad2-ab6f-4a70-8938-fe432207d893","Type":"ContainerDied","Data":"658b837595928cc4da2922df627a2869a6b06cd3c625f75828407e5a7360fc56"}
Jul 17 14:31:10.843898 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.843897    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"
Jul 17 14:31:10.844141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.843925    2575 scope.go:117] "RemoveContainer" containerID="561fa10fe452e3860db69d0a258402a0304a897ce35974cbfbd0ef1e92c5fa6d"
Jul 17 14:31:10.866920 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.866879    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"]
Jul 17 14:31:10.869305 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:10.869262    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-66rvw"]
Jul 17 14:31:12.074783 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.074753    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" path="/var/lib/kubelet/pods/5d245ad2-ab6f-4a70-8938-fe432207d893/volumes"
Jul 17 14:31:12.445969 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.445875    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"]
Jul 17 14:31:12.446350 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446333    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446352    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446365    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:12.446445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446373    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:12.446445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446401    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:12.446445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446421    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:12.446445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446433    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446441    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446460    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446469    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446582    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446593    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:12.446779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446602    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:12.446779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446615    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.446779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.446625    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d245ad2-ab6f-4a70-8938-fe432207d893" containerName="revoke-keys"
Jul 17 14:31:12.451265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.451243    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"
Jul 17 14:31:12.453844 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.453825    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:31:12.459649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.459626    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"]
Jul 17 14:31:12.643126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.643088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb5c4-d380-4e9a-bd87-64761dc394c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw\" (UID: \"2b6cb5c4-d380-4e9a-bd87-64761dc394c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"
Jul 17 14:31:12.743564 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.743537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb5c4-d380-4e9a-bd87-64761dc394c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw\" (UID: \"2b6cb5c4-d380-4e9a-bd87-64761dc394c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"
Jul 17 14:31:12.744188 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.744159    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb5c4-d380-4e9a-bd87-64761dc394c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw\" (UID: \"2b6cb5c4-d380-4e9a-bd87-64761dc394c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"
Jul 17 14:31:12.761998 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.761978    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"
Jul 17 14:31:12.895333 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:12.895249    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"]
Jul 17 14:31:12.897599 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:31:12.897568    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6cb5c4_d380_4e9a_bd87_64761dc394c4.slice/crio-e8ae229555fdd09978e7deb359b2e2989299a80162fef3e8280a1d559f1c3998 WatchSource:0}: Error finding container e8ae229555fdd09978e7deb359b2e2989299a80162fef3e8280a1d559f1c3998: Status 404 returned error can't find the container with id e8ae229555fdd09978e7deb359b2e2989299a80162fef3e8280a1d559f1c3998
Jul 17 14:31:13.861651 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:13.861583    2575 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerID="ce51ce503bcaf5c315756f0e275055dc3e318cf66d0d5aefbcee264b6c25cfca" exitCode=6
Jul 17 14:31:13.861651 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:13.861634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" event={"ID":"2b6cb5c4-d380-4e9a-bd87-64761dc394c4","Type":"ContainerDied","Data":"ce51ce503bcaf5c315756f0e275055dc3e318cf66d0d5aefbcee264b6c25cfca"}
Jul 17 14:31:13.862045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:13.861680    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" event={"ID":"2b6cb5c4-d380-4e9a-bd87-64761dc394c4","Type":"ContainerStarted","Data":"e8ae229555fdd09978e7deb359b2e2989299a80162fef3e8280a1d559f1c3998"}
Jul 17 14:31:13.862045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:13.861831    2575 scope.go:117] "RemoveContainer" containerID="ce51ce503bcaf5c315756f0e275055dc3e318cf66d0d5aefbcee264b6c25cfca"
Jul 17 14:31:14.357305 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.357260    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"]
Jul 17 14:31:14.357670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.357657    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:14.357670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.357671    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:14.357810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.357728    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bface8d-5522-4723-8adb-2acfa31f9ed5" containerName="revoke-keys"
Jul 17 14:31:14.360811 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.360791    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.363258 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.363215    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f387e370-serving-cert\""
Jul 17 14:31:14.374407 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.374384    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"]
Jul 17 14:31:14.456748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.456717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-openshift-service-ca\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.456748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.456750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnn4g\" (UniqueName: \"kubernetes.io/projected/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-kube-api-access-cnn4g\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.456951 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.456768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-maas-api-tls\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.557411 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.557370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-openshift-service-ca\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.557411 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.557413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnn4g\" (UniqueName: \"kubernetes.io/projected/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-kube-api-access-cnn4g\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.557630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.557438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-maas-api-tls\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.558029 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.558002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-openshift-service-ca\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.559903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.559881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-maas-api-tls\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.567706 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.567680    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnn4g\" (UniqueName: \"kubernetes.io/projected/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-kube-api-access-cnn4g\") pod \"maas-api-e2e-mt-f387e370-79df698445-bf72d\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.671922 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.671867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:14.803822 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.803794    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"]
Jul 17 14:31:14.806432 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:31:14.806408    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ae3cce_35dc_4fdb_a740_c4fd431d1ede.slice/crio-697d74942ab901b01dbb016c03dcf740cfe8d51ad0d60460a142feeb4e831c33 WatchSource:0}: Error finding container 697d74942ab901b01dbb016c03dcf740cfe8d51ad0d60460a142feeb4e831c33: Status 404 returned error can't find the container with id 697d74942ab901b01dbb016c03dcf740cfe8d51ad0d60460a142feeb4e831c33
Jul 17 14:31:14.867591 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.867554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d" event={"ID":"16ae3cce-35dc-4fdb-a740-c4fd431d1ede","Type":"ContainerStarted","Data":"697d74942ab901b01dbb016c03dcf740cfe8d51ad0d60460a142feeb4e831c33"}
Jul 17 14:31:14.869691 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.869668    2575 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerID="cb92d5216114ffeebf20e70cda657cf05781fbf7bb3a5cddf4875f8e0b18ae01" exitCode=6
Jul 17 14:31:14.869783 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.869741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" event={"ID":"2b6cb5c4-d380-4e9a-bd87-64761dc394c4","Type":"ContainerDied","Data":"cb92d5216114ffeebf20e70cda657cf05781fbf7bb3a5cddf4875f8e0b18ae01"}
Jul 17 14:31:14.869823 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.869795    2575 scope.go:117] "RemoveContainer" containerID="ce51ce503bcaf5c315756f0e275055dc3e318cf66d0d5aefbcee264b6c25cfca"
Jul 17 14:31:14.870024 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:14.870003    2575 scope.go:117] "RemoveContainer" containerID="cb92d5216114ffeebf20e70cda657cf05781fbf7bb3a5cddf4875f8e0b18ae01"
Jul 17 14:31:14.870240 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:14.870214    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw_odh-ai-gateway-infra(2b6cb5c4-d380-4e9a-bd87-64761dc394c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4"
Jul 17 14:31:15.875769 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:15.875728    2575 scope.go:117] "RemoveContainer" containerID="cb92d5216114ffeebf20e70cda657cf05781fbf7bb3a5cddf4875f8e0b18ae01"
Jul 17 14:31:15.876200 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:15.875929    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw_odh-ai-gateway-infra(2b6cb5c4-d380-4e9a-bd87-64761dc394c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4"
Jul 17 14:31:15.876984 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:15.876962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d" event={"ID":"16ae3cce-35dc-4fdb-a740-c4fd431d1ede","Type":"ContainerStarted","Data":"979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597"}
Jul 17 14:31:15.877122 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:15.877104    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:15.913107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:15.913063    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d" podStartSLOduration=1.624743022 podStartE2EDuration="1.913049892s" totalImagesPullingTime="288.30687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:31:15.911879508 +0000 UTC m=+2754.299226946" watchObservedRunningTime="2026-07-17 14:31:15.913049892 +0000 UTC m=+2754.300397330"
Jul 17 14:31:21.887219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:21.887191    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:31:23.287727 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.287685    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"]
Jul 17 14:31:23.292698 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.292677    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"
Jul 17 14:31:23.298775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.298750    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"]
Jul 17 14:31:23.326162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.326139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d697970-f5cd-4be2-9d5b-2d1f179d889d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t\" (UID: \"1d697970-f5cd-4be2-9d5b-2d1f179d889d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"
Jul 17 14:31:23.427211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.427191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d697970-f5cd-4be2-9d5b-2d1f179d889d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t\" (UID: \"1d697970-f5cd-4be2-9d5b-2d1f179d889d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"
Jul 17 14:31:23.427775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.427758    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d697970-f5cd-4be2-9d5b-2d1f179d889d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t\" (UID: \"1d697970-f5cd-4be2-9d5b-2d1f179d889d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"
Jul 17 14:31:23.605199 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.605133    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"
Jul 17 14:31:23.738141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.738103    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"]
Jul 17 14:31:23.739230 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:31:23.739200    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d697970_f5cd_4be2_9d5b_2d1f179d889d.slice/crio-1ae9355c315bef80e2d02dac717579aa42da904fe4441bab4d2e84dd5cc4d343 WatchSource:0}: Error finding container 1ae9355c315bef80e2d02dac717579aa42da904fe4441bab4d2e84dd5cc4d343: Status 404 returned error can't find the container with id 1ae9355c315bef80e2d02dac717579aa42da904fe4441bab4d2e84dd5cc4d343
Jul 17 14:31:23.915292 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:23.915208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" event={"ID":"1d697970-f5cd-4be2-9d5b-2d1f179d889d","Type":"ContainerStarted","Data":"1ae9355c315bef80e2d02dac717579aa42da904fe4441bab4d2e84dd5cc4d343"}
Jul 17 14:31:24.921966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:24.921928    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerID="af6b654abe08408318ba8625361d3e8319a098e37ce209985a9a0e54e92a3d63" exitCode=6
Jul 17 14:31:24.922369 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:24.922012    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" event={"ID":"1d697970-f5cd-4be2-9d5b-2d1f179d889d","Type":"ContainerDied","Data":"af6b654abe08408318ba8625361d3e8319a098e37ce209985a9a0e54e92a3d63"}
Jul 17 14:31:24.922369 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:24.922223    2575 scope.go:117] "RemoveContainer" containerID="af6b654abe08408318ba8625361d3e8319a098e37ce209985a9a0e54e92a3d63"
Jul 17 14:31:25.928014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:25.927983    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerID="db55bf07750c13e7c64aca3ee1acf5f0419bcf37e949b1b2bf450f79e9289e8e" exitCode=6
Jul 17 14:31:25.928404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:25.928066    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" event={"ID":"1d697970-f5cd-4be2-9d5b-2d1f179d889d","Type":"ContainerDied","Data":"db55bf07750c13e7c64aca3ee1acf5f0419bcf37e949b1b2bf450f79e9289e8e"}
Jul 17 14:31:25.928404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:25.928126    2575 scope.go:117] "RemoveContainer" containerID="af6b654abe08408318ba8625361d3e8319a098e37ce209985a9a0e54e92a3d63"
Jul 17 14:31:25.928496 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:25.928411    2575 scope.go:117] "RemoveContainer" containerID="db55bf07750c13e7c64aca3ee1acf5f0419bcf37e949b1b2bf450f79e9289e8e"
Jul 17 14:31:25.928642 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:25.928625    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t_odh-ai-gateway-infra(1d697970-f5cd-4be2-9d5b-2d1f179d889d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d"
Jul 17 14:31:26.934778 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:26.934749    2575 scope.go:117] "RemoveContainer" containerID="db55bf07750c13e7c64aca3ee1acf5f0419bcf37e949b1b2bf450f79e9289e8e"
Jul 17 14:31:26.935150 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:26.934946    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t_odh-ai-gateway-infra(1d697970-f5cd-4be2-9d5b-2d1f179d889d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d"
Jul 17 14:31:27.068901 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:27.068871    2575 scope.go:117] "RemoveContainer" containerID="cb92d5216114ffeebf20e70cda657cf05781fbf7bb3a5cddf4875f8e0b18ae01"
Jul 17 14:31:27.940552 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:27.940522    2575 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerID="68e434db0e59d211048b0b70839d4a6c563e2e29b0bfaa7e0663eedb2bf7f842" exitCode=6
Jul 17 14:31:27.940858 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:27.940589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" event={"ID":"2b6cb5c4-d380-4e9a-bd87-64761dc394c4","Type":"ContainerDied","Data":"68e434db0e59d211048b0b70839d4a6c563e2e29b0bfaa7e0663eedb2bf7f842"}
Jul 17 14:31:27.940858 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:27.940645    2575 scope.go:117] "RemoveContainer" containerID="cb92d5216114ffeebf20e70cda657cf05781fbf7bb3a5cddf4875f8e0b18ae01"
Jul 17 14:31:27.940946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:27.940887    2575 scope.go:117] "RemoveContainer" containerID="68e434db0e59d211048b0b70839d4a6c563e2e29b0bfaa7e0663eedb2bf7f842"
Jul 17 14:31:27.941125 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:27.941104    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw_odh-ai-gateway-infra(2b6cb5c4-d380-4e9a-bd87-64761dc394c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4"
Jul 17 14:31:28.972114 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:28.972079    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"]
Jul 17 14:31:29.104641 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.104617    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"
Jul 17 14:31:29.176452 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.176423    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb5c4-d380-4e9a-bd87-64761dc394c4-maas-api-service-ca\") pod \"2b6cb5c4-d380-4e9a-bd87-64761dc394c4\" (UID: \"2b6cb5c4-d380-4e9a-bd87-64761dc394c4\") "
Jul 17 14:31:29.176771 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.176749    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6cb5c4-d380-4e9a-bd87-64761dc394c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b6cb5c4-d380-4e9a-bd87-64761dc394c4" (UID: "2b6cb5c4-d380-4e9a-bd87-64761dc394c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:31:29.277184 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.277125    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb5c4-d380-4e9a-bd87-64761dc394c4-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:31:29.952980 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.952952    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"
Jul 17 14:31:29.953155 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.952984    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw" event={"ID":"2b6cb5c4-d380-4e9a-bd87-64761dc394c4","Type":"ContainerDied","Data":"e8ae229555fdd09978e7deb359b2e2989299a80162fef3e8280a1d559f1c3998"}
Jul 17 14:31:29.953155 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.953041    2575 scope.go:117] "RemoveContainer" containerID="68e434db0e59d211048b0b70839d4a6c563e2e29b0bfaa7e0663eedb2bf7f842"
Jul 17 14:31:29.975908 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.975880    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"]
Jul 17 14:31:29.979897 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:29.979872    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mx7rw"]
Jul 17 14:31:30.074025 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:30.074003    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" path="/var/lib/kubelet/pods/2b6cb5c4-d380-4e9a-bd87-64761dc394c4/volumes"
Jul 17 14:31:39.144104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144071    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"]
Jul 17 14:31:39.144463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144451    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:39.144463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144463    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:39.144550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144478    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:39.144550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144484    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:39.144550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144547    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:39.144647 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144555    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:39.144647 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.144563    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:39.147699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.147683    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"
Jul 17 14:31:39.159458 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.159433    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"]
Jul 17 14:31:39.257367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.257338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9145b3-3771-4f9d-91b2-7ececfff3abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w\" (UID: \"3d9145b3-3771-4f9d-91b2-7ececfff3abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"
Jul 17 14:31:39.358165 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.358138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9145b3-3771-4f9d-91b2-7ececfff3abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w\" (UID: \"3d9145b3-3771-4f9d-91b2-7ececfff3abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"
Jul 17 14:31:39.358708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.358690    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9145b3-3771-4f9d-91b2-7ececfff3abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w\" (UID: \"3d9145b3-3771-4f9d-91b2-7ececfff3abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"
Jul 17 14:31:39.457551 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.457496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"
Jul 17 14:31:39.593211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:39.593169    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"]
Jul 17 14:31:39.599450 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:31:39.599426    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9145b3_3771_4f9d_91b2_7ececfff3abf.slice/crio-a9cc9cb28bf837e0456f3382a33ca067bfa2e6aabe655abe3b03096ca3b826fb WatchSource:0}: Error finding container a9cc9cb28bf837e0456f3382a33ca067bfa2e6aabe655abe3b03096ca3b826fb: Status 404 returned error can't find the container with id a9cc9cb28bf837e0456f3382a33ca067bfa2e6aabe655abe3b03096ca3b826fb
Jul 17 14:31:40.001675 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:40.001643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" event={"ID":"3d9145b3-3771-4f9d-91b2-7ececfff3abf","Type":"ContainerStarted","Data":"651bc71c232d8c7c79bb7b21a9d1537c52e25664b19d5d54a02d8521d9a584df"}
Jul 17 14:31:40.001870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:40.001695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" event={"ID":"3d9145b3-3771-4f9d-91b2-7ececfff3abf","Type":"ContainerStarted","Data":"a9cc9cb28bf837e0456f3382a33ca067bfa2e6aabe655abe3b03096ca3b826fb"}
Jul 17 14:31:40.018649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:40.018606    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" podStartSLOduration=1.018592383 podStartE2EDuration="1.018592383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:31:40.017955802 +0000 UTC m=+2778.405303259" watchObservedRunningTime="2026-07-17 14:31:40.018592383 +0000 UTC m=+2778.405939869"
Jul 17 14:31:40.068797 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:40.068735    2575 scope.go:117] "RemoveContainer" containerID="db55bf07750c13e7c64aca3ee1acf5f0419bcf37e949b1b2bf450f79e9289e8e"
Jul 17 14:31:41.008089 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.008056    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerID="5d07e6db6ec21561b8f4ba6326a98ef7073581048393980adf804a9d79a192d6" exitCode=6
Jul 17 14:31:41.008512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.008129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" event={"ID":"1d697970-f5cd-4be2-9d5b-2d1f179d889d","Type":"ContainerDied","Data":"5d07e6db6ec21561b8f4ba6326a98ef7073581048393980adf804a9d79a192d6"}
Jul 17 14:31:41.008512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.008189    2575 scope.go:117] "RemoveContainer" containerID="db55bf07750c13e7c64aca3ee1acf5f0419bcf37e949b1b2bf450f79e9289e8e"
Jul 17 14:31:41.008512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.008497    2575 scope.go:117] "RemoveContainer" containerID="5d07e6db6ec21561b8f4ba6326a98ef7073581048393980adf804a9d79a192d6"
Jul 17 14:31:41.008725 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:41.008697    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t_odh-ai-gateway-infra(1d697970-f5cd-4be2-9d5b-2d1f179d889d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d"
Jul 17 14:31:41.009847 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.009824    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerID="651bc71c232d8c7c79bb7b21a9d1537c52e25664b19d5d54a02d8521d9a584df" exitCode=6
Jul 17 14:31:41.009928 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.009872    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" event={"ID":"3d9145b3-3771-4f9d-91b2-7ececfff3abf","Type":"ContainerDied","Data":"651bc71c232d8c7c79bb7b21a9d1537c52e25664b19d5d54a02d8521d9a584df"}
Jul 17 14:31:41.010045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.010034    2575 scope.go:117] "RemoveContainer" containerID="651bc71c232d8c7c79bb7b21a9d1537c52e25664b19d5d54a02d8521d9a584df"
Jul 17 14:31:41.765220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.765187    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"]
Jul 17 14:31:41.765648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.765634    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:41.765708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.765652    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6cb5c4-d380-4e9a-bd87-64761dc394c4" containerName="revoke-keys"
Jul 17 14:31:41.768756 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.768740    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"
Jul 17 14:31:41.782245 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.782216    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"]
Jul 17 14:31:41.881139 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.881078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1262d825-53be-4796-aa15-d1ab5d2615cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-krfs4\" (UID: \"1262d825-53be-4796-aa15-d1ab5d2615cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"
Jul 17 14:31:41.982382 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.982353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1262d825-53be-4796-aa15-d1ab5d2615cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-krfs4\" (UID: \"1262d825-53be-4796-aa15-d1ab5d2615cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"
Jul 17 14:31:41.982953 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:41.982930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1262d825-53be-4796-aa15-d1ab5d2615cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-krfs4\" (UID: \"1262d825-53be-4796-aa15-d1ab5d2615cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"
Jul 17 14:31:42.016217 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.016188    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerID="4d536d3ed0443670cac1be0c0dddf5d94503111176a6822f29409b05d1412ca3" exitCode=6
Jul 17 14:31:42.016587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.016270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" event={"ID":"3d9145b3-3771-4f9d-91b2-7ececfff3abf","Type":"ContainerDied","Data":"4d536d3ed0443670cac1be0c0dddf5d94503111176a6822f29409b05d1412ca3"}
Jul 17 14:31:42.016587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.016345    2575 scope.go:117] "RemoveContainer" containerID="651bc71c232d8c7c79bb7b21a9d1537c52e25664b19d5d54a02d8521d9a584df"
Jul 17 14:31:42.016587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.016566    2575 scope.go:117] "RemoveContainer" containerID="4d536d3ed0443670cac1be0c0dddf5d94503111176a6822f29409b05d1412ca3"
Jul 17 14:31:42.016789 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:42.016764    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w_odh-ai-gateway-infra(3d9145b3-3771-4f9d-91b2-7ececfff3abf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf"
Jul 17 14:31:42.044371 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.044347    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"]
Jul 17 14:31:42.077921 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.077889    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"
Jul 17 14:31:42.196751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.196726    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"
Jul 17 14:31:42.215094 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.215073    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"]
Jul 17 14:31:42.217725 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:31:42.217685    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1262d825_53be_4796_aa15_d1ab5d2615cc.slice/crio-7e6bc3436b26fc93278adfd10d9c515fdb8d5a84ca30e1d1d04103767f9fd72b WatchSource:0}: Error finding container 7e6bc3436b26fc93278adfd10d9c515fdb8d5a84ca30e1d1d04103767f9fd72b: Status 404 returned error can't find the container with id 7e6bc3436b26fc93278adfd10d9c515fdb8d5a84ca30e1d1d04103767f9fd72b
Jul 17 14:31:42.285240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.285211    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d697970-f5cd-4be2-9d5b-2d1f179d889d-maas-api-service-ca\") pod \"1d697970-f5cd-4be2-9d5b-2d1f179d889d\" (UID: \"1d697970-f5cd-4be2-9d5b-2d1f179d889d\") "
Jul 17 14:31:42.285562 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.285537    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d697970-f5cd-4be2-9d5b-2d1f179d889d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d697970-f5cd-4be2-9d5b-2d1f179d889d" (UID: "1d697970-f5cd-4be2-9d5b-2d1f179d889d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:31:42.386468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:42.386415    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d697970-f5cd-4be2-9d5b-2d1f179d889d-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:31:43.024326 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.024224    2575 scope.go:117] "RemoveContainer" containerID="4d536d3ed0443670cac1be0c0dddf5d94503111176a6822f29409b05d1412ca3"
Jul 17 14:31:43.024813 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:43.024482    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w_odh-ai-gateway-infra(3d9145b3-3771-4f9d-91b2-7ececfff3abf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf"
Jul 17 14:31:43.025256 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.025238    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"
Jul 17 14:31:43.025343 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.025265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t" event={"ID":"1d697970-f5cd-4be2-9d5b-2d1f179d889d","Type":"ContainerDied","Data":"1ae9355c315bef80e2d02dac717579aa42da904fe4441bab4d2e84dd5cc4d343"}
Jul 17 14:31:43.025343 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.025348    2575 scope.go:117] "RemoveContainer" containerID="5d07e6db6ec21561b8f4ba6326a98ef7073581048393980adf804a9d79a192d6"
Jul 17 14:31:43.026865 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.026840    2575 generic.go:356] "Generic (PLEG): container finished" podID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerID="0a91c48f1cbaaaed1fd65754f954e739cd7f6a171fba65302bdc904b328b8789" exitCode=6
Jul 17 14:31:43.026984 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.026862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" event={"ID":"1262d825-53be-4796-aa15-d1ab5d2615cc","Type":"ContainerDied","Data":"0a91c48f1cbaaaed1fd65754f954e739cd7f6a171fba65302bdc904b328b8789"}
Jul 17 14:31:43.026984 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.026914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" event={"ID":"1262d825-53be-4796-aa15-d1ab5d2615cc","Type":"ContainerStarted","Data":"7e6bc3436b26fc93278adfd10d9c515fdb8d5a84ca30e1d1d04103767f9fd72b"}
Jul 17 14:31:43.027095 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.027070    2575 scope.go:117] "RemoveContainer" containerID="0a91c48f1cbaaaed1fd65754f954e739cd7f6a171fba65302bdc904b328b8789"
Jul 17 14:31:43.064018 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.063985    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"]
Jul 17 14:31:43.069172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:43.069150    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rpv6t"]
Jul 17 14:31:44.034673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:44.034633    2575 generic.go:356] "Generic (PLEG): container finished" podID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerID="1888c86cd3d93db37120799082af95e3da6b3ee930671574eb9675160358802e" exitCode=6
Jul 17 14:31:44.034673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:44.034666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" event={"ID":"1262d825-53be-4796-aa15-d1ab5d2615cc","Type":"ContainerDied","Data":"1888c86cd3d93db37120799082af95e3da6b3ee930671574eb9675160358802e"}
Jul 17 14:31:44.035175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:44.034748    2575 scope.go:117] "RemoveContainer" containerID="0a91c48f1cbaaaed1fd65754f954e739cd7f6a171fba65302bdc904b328b8789"
Jul 17 14:31:44.035175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:44.034945    2575 scope.go:117] "RemoveContainer" containerID="1888c86cd3d93db37120799082af95e3da6b3ee930671574eb9675160358802e"
Jul 17 14:31:44.035272 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:44.035181    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-krfs4_odh-ai-gateway-infra(1262d825-53be-4796-aa15-d1ab5d2615cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc"
Jul 17 14:31:44.074707 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:44.074673    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" path="/var/lib/kubelet/pods/1d697970-f5cd-4be2-9d5b-2d1f179d889d/volumes"
Jul 17 14:31:45.041868 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:45.041843    2575 scope.go:117] "RemoveContainer" containerID="1888c86cd3d93db37120799082af95e3da6b3ee930671574eb9675160358802e"
Jul 17 14:31:45.042230 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:45.042012    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-krfs4_odh-ai-gateway-infra(1262d825-53be-4796-aa15-d1ab5d2615cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc"
Jul 17 14:31:47.025671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.025639    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"]
Jul 17 14:31:47.026066 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.026031    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:31:47.026066 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.026042    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:31:47.026066 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.026061    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:31:47.026066 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.026066    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:31:47.026209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.026117    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:31:47.026209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.026128    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:31:47.030582 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.030566    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"
Jul 17 14:31:47.040635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.040613    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"]
Jul 17 14:31:47.129216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.129174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b22ea-6107-4b39-bcc1-8c522d15a313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f387e370-8m5pn\" (UID: \"496b22ea-6107-4b39-bcc1-8c522d15a313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"
Jul 17 14:31:47.230531 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.230498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b22ea-6107-4b39-bcc1-8c522d15a313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f387e370-8m5pn\" (UID: \"496b22ea-6107-4b39-bcc1-8c522d15a313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"
Jul 17 14:31:47.231137 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.231112    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b22ea-6107-4b39-bcc1-8c522d15a313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f387e370-8m5pn\" (UID: \"496b22ea-6107-4b39-bcc1-8c522d15a313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"
Jul 17 14:31:47.341862 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.341778    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"
Jul 17 14:31:47.495955 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:47.495922    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"]
Jul 17 14:31:47.497203 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:31:47.497169    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496b22ea_6107_4b39_bcc1_8c522d15a313.slice/crio-366ee16144b35f9a0239f6b0ec63f2419e4650e52f660b9f90495af00e4c500a WatchSource:0}: Error finding container 366ee16144b35f9a0239f6b0ec63f2419e4650e52f660b9f90495af00e4c500a: Status 404 returned error can't find the container with id 366ee16144b35f9a0239f6b0ec63f2419e4650e52f660b9f90495af00e4c500a
Jul 17 14:31:48.057759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:48.057682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn" event={"ID":"496b22ea-6107-4b39-bcc1-8c522d15a313","Type":"ContainerStarted","Data":"3b001632ff92a23df8bcb6940b5dde064ac715b5ecefde942da501c492f8d36f"}
Jul 17 14:31:48.057759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:48.057734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn" event={"ID":"496b22ea-6107-4b39-bcc1-8c522d15a313","Type":"ContainerStarted","Data":"366ee16144b35f9a0239f6b0ec63f2419e4650e52f660b9f90495af00e4c500a"}
Jul 17 14:31:48.075655 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:48.075612    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn" podStartSLOduration=1.0755973540000001 podStartE2EDuration="1.075597354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:31:48.074930986 +0000 UTC m=+2786.462278426" watchObservedRunningTime="2026-07-17 14:31:48.075597354 +0000 UTC m=+2786.462944794"
Jul 17 14:31:49.065244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:49.065213    2575 generic.go:356] "Generic (PLEG): container finished" podID="496b22ea-6107-4b39-bcc1-8c522d15a313" containerID="3b001632ff92a23df8bcb6940b5dde064ac715b5ecefde942da501c492f8d36f" exitCode=0
Jul 17 14:31:49.065605 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:49.065261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn" event={"ID":"496b22ea-6107-4b39-bcc1-8c522d15a313","Type":"ContainerDied","Data":"3b001632ff92a23df8bcb6940b5dde064ac715b5ecefde942da501c492f8d36f"}
Jul 17 14:31:50.198696 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:50.198672    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"
Jul 17 14:31:50.355575 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:50.355498    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b22ea-6107-4b39-bcc1-8c522d15a313-maas-api-service-ca\") pod \"496b22ea-6107-4b39-bcc1-8c522d15a313\" (UID: \"496b22ea-6107-4b39-bcc1-8c522d15a313\") "
Jul 17 14:31:50.355827 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:50.355806    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496b22ea-6107-4b39-bcc1-8c522d15a313-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "496b22ea-6107-4b39-bcc1-8c522d15a313" (UID: "496b22ea-6107-4b39-bcc1-8c522d15a313"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:31:50.456508 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:50.456473    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b22ea-6107-4b39-bcc1-8c522d15a313-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:31:51.076736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:51.076704    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn" event={"ID":"496b22ea-6107-4b39-bcc1-8c522d15a313","Type":"ContainerDied","Data":"366ee16144b35f9a0239f6b0ec63f2419e4650e52f660b9f90495af00e4c500a"}
Jul 17 14:31:51.076736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:51.076725    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"
Jul 17 14:31:51.076930 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:51.076750    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="366ee16144b35f9a0239f6b0ec63f2419e4650e52f660b9f90495af00e4c500a"
Jul 17 14:31:54.069217 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:54.069186    2575 scope.go:117] "RemoveContainer" containerID="4d536d3ed0443670cac1be0c0dddf5d94503111176a6822f29409b05d1412ca3"
Jul 17 14:31:55.098375 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:55.098345    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerID="d503c8cbfe7f47dca22306af6af0da03c67f3f5917885e939dc882a7a3958028" exitCode=6
Jul 17 14:31:55.098779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:55.098389    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" event={"ID":"3d9145b3-3771-4f9d-91b2-7ececfff3abf","Type":"ContainerDied","Data":"d503c8cbfe7f47dca22306af6af0da03c67f3f5917885e939dc882a7a3958028"}
Jul 17 14:31:55.098779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:55.098430    2575 scope.go:117] "RemoveContainer" containerID="4d536d3ed0443670cac1be0c0dddf5d94503111176a6822f29409b05d1412ca3"
Jul 17 14:31:55.098779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:55.098709    2575 scope.go:117] "RemoveContainer" containerID="d503c8cbfe7f47dca22306af6af0da03c67f3f5917885e939dc882a7a3958028"
Jul 17 14:31:55.098948 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:31:55.098898    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w_odh-ai-gateway-infra(3d9145b3-3771-4f9d-91b2-7ececfff3abf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf"
Jul 17 14:31:56.127522 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:56.127490    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"]
Jul 17 14:31:56.260597 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:56.260575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"
Jul 17 14:31:56.404789 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:56.404701    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9145b3-3771-4f9d-91b2-7ececfff3abf-maas-api-service-ca\") pod \"3d9145b3-3771-4f9d-91b2-7ececfff3abf\" (UID: \"3d9145b3-3771-4f9d-91b2-7ececfff3abf\") "
Jul 17 14:31:56.405037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:56.405013    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9145b3-3771-4f9d-91b2-7ececfff3abf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d9145b3-3771-4f9d-91b2-7ececfff3abf" (UID: "3d9145b3-3771-4f9d-91b2-7ececfff3abf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:31:56.505496 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:56.505467    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9145b3-3771-4f9d-91b2-7ececfff3abf-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:31:57.111393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:57.111361    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"
Jul 17 14:31:57.111393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:57.111376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w" event={"ID":"3d9145b3-3771-4f9d-91b2-7ececfff3abf","Type":"ContainerDied","Data":"a9cc9cb28bf837e0456f3382a33ca067bfa2e6aabe655abe3b03096ca3b826fb"}
Jul 17 14:31:57.111611 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:57.111426    2575 scope.go:117] "RemoveContainer" containerID="d503c8cbfe7f47dca22306af6af0da03c67f3f5917885e939dc882a7a3958028"
Jul 17 14:31:57.134475 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:57.134446    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"]
Jul 17 14:31:57.140384 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:57.140362    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-xlk9w"]
Jul 17 14:31:58.074387 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:31:58.074358    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" path="/var/lib/kubelet/pods/3d9145b3-3771-4f9d-91b2-7ececfff3abf/volumes"
Jul 17 14:32:00.068997 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:00.068970    2575 scope.go:117] "RemoveContainer" containerID="1888c86cd3d93db37120799082af95e3da6b3ee930671574eb9675160358802e"
Jul 17 14:32:01.133594 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:01.133560    2575 generic.go:356] "Generic (PLEG): container finished" podID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerID="11f9f24e1c355be691a37f220168b324d677aff34b205a89107381d1ce1deb89" exitCode=6
Jul 17 14:32:01.133995 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:01.133630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" event={"ID":"1262d825-53be-4796-aa15-d1ab5d2615cc","Type":"ContainerDied","Data":"11f9f24e1c355be691a37f220168b324d677aff34b205a89107381d1ce1deb89"}
Jul 17 14:32:01.133995 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:01.133711    2575 scope.go:117] "RemoveContainer" containerID="1888c86cd3d93db37120799082af95e3da6b3ee930671574eb9675160358802e"
Jul 17 14:32:01.134149 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:01.133993    2575 scope.go:117] "RemoveContainer" containerID="11f9f24e1c355be691a37f220168b324d677aff34b205a89107381d1ce1deb89"
Jul 17 14:32:01.134259 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:01.134219    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-krfs4_odh-ai-gateway-infra(1262d825-53be-4796-aa15-d1ab5d2615cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc"
Jul 17 14:32:02.161517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.161487    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"]
Jul 17 14:32:02.279817 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.279793    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"
Jul 17 14:32:02.452594 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.452530    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1262d825-53be-4796-aa15-d1ab5d2615cc-maas-api-service-ca\") pod \"1262d825-53be-4796-aa15-d1ab5d2615cc\" (UID: \"1262d825-53be-4796-aa15-d1ab5d2615cc\") "
Jul 17 14:32:02.452850 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.452831    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1262d825-53be-4796-aa15-d1ab5d2615cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1262d825-53be-4796-aa15-d1ab5d2615cc" (UID: "1262d825-53be-4796-aa15-d1ab5d2615cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:32:02.500581 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.500545    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"]
Jul 17 14:32:02.501098 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501083    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501102    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501114    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:02.501142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501123    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:02.501142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501135    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501143    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501153    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496b22ea-6107-4b39-bcc1-8c522d15a313" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501161    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="496b22ea-6107-4b39-bcc1-8c522d15a313" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501181    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501189    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501203    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501211    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501226    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:02.501265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501233    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:02.501538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501323    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d697970-f5cd-4be2-9d5b-2d1f179d889d" containerName="revoke-keys"
Jul 17 14:32:02.501538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501337    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501346    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:02.501538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501354    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:02.501538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501364    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:02.501538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.501373    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="496b22ea-6107-4b39-bcc1-8c522d15a313" containerName="revoke-keys"
Jul 17 14:32:02.504326 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.504309    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"
Jul 17 14:32:02.513606 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.513579    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"]
Jul 17 14:32:02.553704 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.553680    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1262d825-53be-4796-aa15-d1ab5d2615cc-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:32:02.575344 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.575317    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"]
Jul 17 14:32:02.575602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.575548    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d" podUID="16ae3cce-35dc-4fdb-a740-c4fd431d1ede" containerName="maas-api" containerID="cri-o://979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597" gracePeriod=30
Jul 17 14:32:02.654730 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.654703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba956fe-68fd-403e-a1a2-40b1f3911e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz\" (UID: \"eba956fe-68fd-403e-a1a2-40b1f3911e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"
Jul 17 14:32:02.755327 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.755259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba956fe-68fd-403e-a1a2-40b1f3911e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz\" (UID: \"eba956fe-68fd-403e-a1a2-40b1f3911e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"
Jul 17 14:32:02.756030 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.756007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba956fe-68fd-403e-a1a2-40b1f3911e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz\" (UID: \"eba956fe-68fd-403e-a1a2-40b1f3911e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"
Jul 17 14:32:02.815729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.815707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"
Jul 17 14:32:02.825007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.824981    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:32:02.946698 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.946667    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"]
Jul 17 14:32:02.947452 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:32:02.947419    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba956fe_68fd_403e_a1a2_40b1f3911e22.slice/crio-9d8cc3ef7e8dddb2fa5be44c8fc541c30315113a9f192d6c8575a7c4fee3000b WatchSource:0}: Error finding container 9d8cc3ef7e8dddb2fa5be44c8fc541c30315113a9f192d6c8575a7c4fee3000b: Status 404 returned error can't find the container with id 9d8cc3ef7e8dddb2fa5be44c8fc541c30315113a9f192d6c8575a7c4fee3000b
Jul 17 14:32:02.956582 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.956561    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnn4g\" (UniqueName: \"kubernetes.io/projected/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-kube-api-access-cnn4g\") pod \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") "
Jul 17 14:32:02.956716 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.956614    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-openshift-service-ca\") pod \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") "
Jul 17 14:32:02.956716 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.956633    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-maas-api-tls\") pod \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\" (UID: \"16ae3cce-35dc-4fdb-a740-c4fd431d1ede\") "
Jul 17 14:32:02.956958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.956933    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16ae3cce-35dc-4fdb-a740-c4fd431d1ede" (UID: "16ae3cce-35dc-4fdb-a740-c4fd431d1ede"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:32:02.958560 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.958534    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-kube-api-access-cnn4g" (OuterVolumeSpecName: "kube-api-access-cnn4g") pod "16ae3cce-35dc-4fdb-a740-c4fd431d1ede" (UID: "16ae3cce-35dc-4fdb-a740-c4fd431d1ede"). InnerVolumeSpecName "kube-api-access-cnn4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:32:02.958633 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:02.958576    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16ae3cce-35dc-4fdb-a740-c4fd431d1ede" (UID: "16ae3cce-35dc-4fdb-a740-c4fd431d1ede"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:32:03.058037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.057988    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:32:03.058037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.058009    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:32:03.058037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.058019    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnn4g\" (UniqueName: \"kubernetes.io/projected/16ae3cce-35dc-4fdb-a740-c4fd431d1ede-kube-api-access-cnn4g\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:32:03.145303 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.145259    2575 generic.go:356] "Generic (PLEG): container finished" podID="16ae3cce-35dc-4fdb-a740-c4fd431d1ede" containerID="979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597" exitCode=0
Jul 17 14:32:03.145407 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.145317    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"
Jul 17 14:32:03.145407 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.145342    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d" event={"ID":"16ae3cce-35dc-4fdb-a740-c4fd431d1ede","Type":"ContainerDied","Data":"979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597"}
Jul 17 14:32:03.145407 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.145391    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d" event={"ID":"16ae3cce-35dc-4fdb-a740-c4fd431d1ede","Type":"ContainerDied","Data":"697d74942ab901b01dbb016c03dcf740cfe8d51ad0d60460a142feeb4e831c33"}
Jul 17 14:32:03.145520 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.145416    2575 scope.go:117] "RemoveContainer" containerID="979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597"
Jul 17 14:32:03.146638 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.146624    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"
Jul 17 14:32:03.146726 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.146677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4" event={"ID":"1262d825-53be-4796-aa15-d1ab5d2615cc","Type":"ContainerDied","Data":"7e6bc3436b26fc93278adfd10d9c515fdb8d5a84ca30e1d1d04103767f9fd72b"}
Jul 17 14:32:03.148094 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.148068    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" event={"ID":"eba956fe-68fd-403e-a1a2-40b1f3911e22","Type":"ContainerStarted","Data":"9d8cc3ef7e8dddb2fa5be44c8fc541c30315113a9f192d6c8575a7c4fee3000b"}
Jul 17 14:32:03.154989 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.154859    2575 scope.go:117] "RemoveContainer" containerID="979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597"
Jul 17 14:32:03.155136 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:03.155113    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597\": container with ID starting with 979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597 not found: ID does not exist" containerID="979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597"
Jul 17 14:32:03.155208 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.155148    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597"} err="failed to get container status \"979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597\": rpc error: code = NotFound desc = could not find container \"979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597\": container with ID starting with 979feab422c67d98a8668ca1602d5775d6d50af0d69a3e7337935387e4f4f597 not found: ID does not exist"
Jul 17 14:32:03.155208 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.155171    2575 scope.go:117] "RemoveContainer" containerID="11f9f24e1c355be691a37f220168b324d677aff34b205a89107381d1ce1deb89"
Jul 17 14:32:03.184220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.184196    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"]
Jul 17 14:32:03.188528 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.188509    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-krfs4"]
Jul 17 14:32:03.200004 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.199883    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"]
Jul 17 14:32:03.201686 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:03.201667    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f387e370-79df698445-bf72d"]
Jul 17 14:32:04.074797 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:04.074767    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" path="/var/lib/kubelet/pods/1262d825-53be-4796-aa15-d1ab5d2615cc/volumes"
Jul 17 14:32:04.075147 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:04.075135    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ae3cce-35dc-4fdb-a740-c4fd431d1ede" path="/var/lib/kubelet/pods/16ae3cce-35dc-4fdb-a740-c4fd431d1ede/volumes"
Jul 17 14:32:04.155328 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:04.155304    2575 generic.go:356] "Generic (PLEG): container finished" podID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerID="c9a7c169ac8d05774d651b05c3088936d92f5f59f411438760af511970755a81" exitCode=6
Jul 17 14:32:04.155469 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:04.155359    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" event={"ID":"eba956fe-68fd-403e-a1a2-40b1f3911e22","Type":"ContainerDied","Data":"c9a7c169ac8d05774d651b05c3088936d92f5f59f411438760af511970755a81"}
Jul 17 14:32:04.155577 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:04.155565    2575 scope.go:117] "RemoveContainer" containerID="c9a7c169ac8d05774d651b05c3088936d92f5f59f411438760af511970755a81"
Jul 17 14:32:05.162865 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:05.162779    2575 generic.go:356] "Generic (PLEG): container finished" podID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerID="cba771a528e91297eae14c47d6d8f1b1114c5b955dddffd4846da7f83d06342d" exitCode=6
Jul 17 14:32:05.162865 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:05.162838    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" event={"ID":"eba956fe-68fd-403e-a1a2-40b1f3911e22","Type":"ContainerDied","Data":"cba771a528e91297eae14c47d6d8f1b1114c5b955dddffd4846da7f83d06342d"}
Jul 17 14:32:05.163330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:05.162923    2575 scope.go:117] "RemoveContainer" containerID="c9a7c169ac8d05774d651b05c3088936d92f5f59f411438760af511970755a81"
Jul 17 14:32:05.163330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:05.163165    2575 scope.go:117] "RemoveContainer" containerID="cba771a528e91297eae14c47d6d8f1b1114c5b955dddffd4846da7f83d06342d"
Jul 17 14:32:05.163447 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:05.163386    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz_odh-ai-gateway-infra(eba956fe-68fd-403e-a1a2-40b1f3911e22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22"
Jul 17 14:32:06.169195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:06.169170    2575 scope.go:117] "RemoveContainer" containerID="cba771a528e91297eae14c47d6d8f1b1114c5b955dddffd4846da7f83d06342d"
Jul 17 14:32:06.169599 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:06.169387    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz_odh-ai-gateway-infra(eba956fe-68fd-403e-a1a2-40b1f3911e22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22"
Jul 17 14:32:07.161165 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:07.161131    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"]
Jul 17 14:32:07.164013 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:07.163988    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f387e370-8m5pn"]
Jul 17 14:32:08.075373 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:08.075345    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496b22ea-6107-4b39-bcc1-8c522d15a313" path="/var/lib/kubelet/pods/496b22ea-6107-4b39-bcc1-8c522d15a313/volumes"
Jul 17 14:32:13.340250 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340215    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"]
Jul 17 14:32:13.340649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340608    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ae3cce-35dc-4fdb-a740-c4fd431d1ede" containerName="maas-api"
Jul 17 14:32:13.340649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340619    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ae3cce-35dc-4fdb-a740-c4fd431d1ede" containerName="maas-api"
Jul 17 14:32:13.340649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340635    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:13.340649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340643    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:13.340786 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340711    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9145b3-3771-4f9d-91b2-7ececfff3abf" containerName="revoke-keys"
Jul 17 14:32:13.340786 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340720    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1262d825-53be-4796-aa15-d1ab5d2615cc" containerName="revoke-keys"
Jul 17 14:32:13.340786 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.340728    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ae3cce-35dc-4fdb-a740-c4fd431d1ede" containerName="maas-api"
Jul 17 14:32:13.345068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.345051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"
Jul 17 14:32:13.356535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.356512    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"]
Jul 17 14:32:13.440888 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.440859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b76dd-dd62-4eaf-9600-bf9c34397b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5\" (UID: \"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"
Jul 17 14:32:13.541666 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.541624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b76dd-dd62-4eaf-9600-bf9c34397b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5\" (UID: \"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"
Jul 17 14:32:13.542194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.542176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b76dd-dd62-4eaf-9600-bf9c34397b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5\" (UID: \"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"
Jul 17 14:32:13.654830 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.654760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"
Jul 17 14:32:13.790570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:13.790540    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"]
Jul 17 14:32:13.793567 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:32:13.793538    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7b76dd_dd62_4eaf_9600_bf9c34397b8e.slice/crio-82c86c98a0623f32c1ff8553e82c3154c1a5fa9f46f3ecf92a2e7eda7eb3a261 WatchSource:0}: Error finding container 82c86c98a0623f32c1ff8553e82c3154c1a5fa9f46f3ecf92a2e7eda7eb3a261: Status 404 returned error can't find the container with id 82c86c98a0623f32c1ff8553e82c3154c1a5fa9f46f3ecf92a2e7eda7eb3a261
Jul 17 14:32:14.208822 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:14.208792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" event={"ID":"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e","Type":"ContainerStarted","Data":"0e42a95cc376a362a1f0ddb72bff03f28e0a46a61f3535b33b3e6c46a43ab724"}
Jul 17 14:32:14.209010 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:14.208840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" event={"ID":"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e","Type":"ContainerStarted","Data":"82c86c98a0623f32c1ff8553e82c3154c1a5fa9f46f3ecf92a2e7eda7eb3a261"}
Jul 17 14:32:14.228354 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:14.228260    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" podStartSLOduration=1.228244072 podStartE2EDuration="1.228244072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:32:14.225932202 +0000 UTC m=+2812.613279641" watchObservedRunningTime="2026-07-17 14:32:14.228244072 +0000 UTC m=+2812.615591511"
Jul 17 14:32:15.216164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:15.216134    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerID="0e42a95cc376a362a1f0ddb72bff03f28e0a46a61f3535b33b3e6c46a43ab724" exitCode=6
Jul 17 14:32:15.216565 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:15.216196    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" event={"ID":"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e","Type":"ContainerDied","Data":"0e42a95cc376a362a1f0ddb72bff03f28e0a46a61f3535b33b3e6c46a43ab724"}
Jul 17 14:32:15.216565 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:15.216398    2575 scope.go:117] "RemoveContainer" containerID="0e42a95cc376a362a1f0ddb72bff03f28e0a46a61f3535b33b3e6c46a43ab724"
Jul 17 14:32:16.222729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:16.222696    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerID="3c3c4cf42e563205373397f79cf7144c3c15a3904e903cce24fd19a412365ab2" exitCode=6
Jul 17 14:32:16.223135 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:16.222732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" event={"ID":"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e","Type":"ContainerDied","Data":"3c3c4cf42e563205373397f79cf7144c3c15a3904e903cce24fd19a412365ab2"}
Jul 17 14:32:16.223135 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:16.222777    2575 scope.go:117] "RemoveContainer" containerID="0e42a95cc376a362a1f0ddb72bff03f28e0a46a61f3535b33b3e6c46a43ab724"
Jul 17 14:32:16.223135 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:16.222973    2575 scope.go:117] "RemoveContainer" containerID="3c3c4cf42e563205373397f79cf7144c3c15a3904e903cce24fd19a412365ab2"
Jul 17 14:32:16.223261 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:16.223198    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5_odh-ai-gateway-infra(3d7b76dd-dd62-4eaf-9600-bf9c34397b8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e"
Jul 17 14:32:17.229294 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:17.229252    2575 scope.go:117] "RemoveContainer" containerID="3c3c4cf42e563205373397f79cf7144c3c15a3904e903cce24fd19a412365ab2"
Jul 17 14:32:17.229682 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:17.229448    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5_odh-ai-gateway-infra(3d7b76dd-dd62-4eaf-9600-bf9c34397b8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e"
Jul 17 14:32:18.068903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:18.068874    2575 scope.go:117] "RemoveContainer" containerID="cba771a528e91297eae14c47d6d8f1b1114c5b955dddffd4846da7f83d06342d"
Jul 17 14:32:19.241611 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:19.241580    2575 generic.go:356] "Generic (PLEG): container finished" podID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerID="6e1ed236dd569d6e888adec6e4ffee9358ac3a3d6f11e44350ef7175f0e662ab" exitCode=6
Jul 17 14:32:19.242041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:19.241650    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" event={"ID":"eba956fe-68fd-403e-a1a2-40b1f3911e22","Type":"ContainerDied","Data":"6e1ed236dd569d6e888adec6e4ffee9358ac3a3d6f11e44350ef7175f0e662ab"}
Jul 17 14:32:19.242041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:19.241723    2575 scope.go:117] "RemoveContainer" containerID="cba771a528e91297eae14c47d6d8f1b1114c5b955dddffd4846da7f83d06342d"
Jul 17 14:32:19.242041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:19.242031    2575 scope.go:117] "RemoveContainer" containerID="6e1ed236dd569d6e888adec6e4ffee9358ac3a3d6f11e44350ef7175f0e662ab"
Jul 17 14:32:19.242245 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:19.242223    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz_odh-ai-gateway-infra(eba956fe-68fd-403e-a1a2-40b1f3911e22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22"
Jul 17 14:32:20.280001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:20.279955    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"]
Jul 17 14:32:20.411291 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:20.411255    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"
Jul 17 14:32:20.499981 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:20.499948    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba956fe-68fd-403e-a1a2-40b1f3911e22-maas-api-service-ca\") pod \"eba956fe-68fd-403e-a1a2-40b1f3911e22\" (UID: \"eba956fe-68fd-403e-a1a2-40b1f3911e22\") "
Jul 17 14:32:20.500296 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:20.500256    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba956fe-68fd-403e-a1a2-40b1f3911e22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eba956fe-68fd-403e-a1a2-40b1f3911e22" (UID: "eba956fe-68fd-403e-a1a2-40b1f3911e22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:32:20.600961 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:20.600886    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba956fe-68fd-403e-a1a2-40b1f3911e22-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:32:21.252997 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:21.252968    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"
Jul 17 14:32:21.253163 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:21.253008    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz" event={"ID":"eba956fe-68fd-403e-a1a2-40b1f3911e22","Type":"ContainerDied","Data":"9d8cc3ef7e8dddb2fa5be44c8fc541c30315113a9f192d6c8575a7c4fee3000b"}
Jul 17 14:32:21.253163 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:21.253068    2575 scope.go:117] "RemoveContainer" containerID="6e1ed236dd569d6e888adec6e4ffee9358ac3a3d6f11e44350ef7175f0e662ab"
Jul 17 14:32:21.278062 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:21.278039    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"]
Jul 17 14:32:21.281679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:21.281659    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-m7stz"]
Jul 17 14:32:22.074792 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:22.074760    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" path="/var/lib/kubelet/pods/eba956fe-68fd-403e-a1a2-40b1f3911e22/volumes"
Jul 17 14:32:29.191113 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191077    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"]
Jul 17 14:32:29.191503 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191451    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:29.191503 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191463    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:29.191503 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191489    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:29.191503 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191496    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:29.191630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191559    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:29.191630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191568    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:29.191630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.191575    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:29.196306 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.196268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"
Jul 17 14:32:29.203697 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.203669    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"]
Jul 17 14:32:29.268228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.268197    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c1fe0b-5527-4673-a2b8-aea4e3b2f126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq\" (UID: \"95c1fe0b-5527-4673-a2b8-aea4e3b2f126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"
Jul 17 14:32:29.368614 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.368587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c1fe0b-5527-4673-a2b8-aea4e3b2f126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq\" (UID: \"95c1fe0b-5527-4673-a2b8-aea4e3b2f126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"
Jul 17 14:32:29.369209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.369190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c1fe0b-5527-4673-a2b8-aea4e3b2f126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq\" (UID: \"95c1fe0b-5527-4673-a2b8-aea4e3b2f126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"
Jul 17 14:32:29.507713 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.507687    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"
Jul 17 14:32:29.849544 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:29.849470    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"]
Jul 17 14:32:29.852746 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:32:29.852723    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c1fe0b_5527_4673_a2b8_aea4e3b2f126.slice/crio-d6c276dede59e5297ce1f46c9276077a34e219f4acf7a4e38641e7bd1287dfad WatchSource:0}: Error finding container d6c276dede59e5297ce1f46c9276077a34e219f4acf7a4e38641e7bd1287dfad: Status 404 returned error can't find the container with id d6c276dede59e5297ce1f46c9276077a34e219f4acf7a4e38641e7bd1287dfad
Jul 17 14:32:30.296600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:30.296571    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" event={"ID":"95c1fe0b-5527-4673-a2b8-aea4e3b2f126","Type":"ContainerStarted","Data":"e68233cb1c86d5bc4e406556b68c5446d4b35bc15778be73f22cec53ec30e55e"}
Jul 17 14:32:30.296979 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:30.296618    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" event={"ID":"95c1fe0b-5527-4673-a2b8-aea4e3b2f126","Type":"ContainerStarted","Data":"d6c276dede59e5297ce1f46c9276077a34e219f4acf7a4e38641e7bd1287dfad"}
Jul 17 14:32:30.331637 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:30.331591    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" podStartSLOduration=1.331577519 podStartE2EDuration="1.331577519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:32:30.330703531 +0000 UTC m=+2828.718050972" watchObservedRunningTime="2026-07-17 14:32:30.331577519 +0000 UTC m=+2828.718924960"
Jul 17 14:32:31.069716 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:31.069683    2575 scope.go:117] "RemoveContainer" containerID="3c3c4cf42e563205373397f79cf7144c3c15a3904e903cce24fd19a412365ab2"
Jul 17 14:32:31.302518 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:31.302492    2575 generic.go:356] "Generic (PLEG): container finished" podID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerID="e68233cb1c86d5bc4e406556b68c5446d4b35bc15778be73f22cec53ec30e55e" exitCode=6
Jul 17 14:32:31.302902 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:31.302572    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" event={"ID":"95c1fe0b-5527-4673-a2b8-aea4e3b2f126","Type":"ContainerDied","Data":"e68233cb1c86d5bc4e406556b68c5446d4b35bc15778be73f22cec53ec30e55e"}
Jul 17 14:32:31.302902 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:31.302747    2575 scope.go:117] "RemoveContainer" containerID="e68233cb1c86d5bc4e406556b68c5446d4b35bc15778be73f22cec53ec30e55e"
Jul 17 14:32:32.308531 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.308499    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerID="11c96efe6eaf95fe5dc3d6ff157b0c9455587b332fdedeb31f6a9782439301a3" exitCode=6
Jul 17 14:32:32.308963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.308572    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" event={"ID":"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e","Type":"ContainerDied","Data":"11c96efe6eaf95fe5dc3d6ff157b0c9455587b332fdedeb31f6a9782439301a3"}
Jul 17 14:32:32.308963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.308630    2575 scope.go:117] "RemoveContainer" containerID="3c3c4cf42e563205373397f79cf7144c3c15a3904e903cce24fd19a412365ab2"
Jul 17 14:32:32.308963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.308883    2575 scope.go:117] "RemoveContainer" containerID="11c96efe6eaf95fe5dc3d6ff157b0c9455587b332fdedeb31f6a9782439301a3"
Jul 17 14:32:32.309156 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:32.309125    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5_odh-ai-gateway-infra(3d7b76dd-dd62-4eaf-9600-bf9c34397b8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e"
Jul 17 14:32:32.310660 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.310639    2575 generic.go:356] "Generic (PLEG): container finished" podID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerID="9ff5912da41decfef611c115a0c18987433df2737d97223b371284e29683366f" exitCode=6
Jul 17 14:32:32.310774 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.310717    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" event={"ID":"95c1fe0b-5527-4673-a2b8-aea4e3b2f126","Type":"ContainerDied","Data":"9ff5912da41decfef611c115a0c18987433df2737d97223b371284e29683366f"}
Jul 17 14:32:32.310908 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.310895    2575 scope.go:117] "RemoveContainer" containerID="9ff5912da41decfef611c115a0c18987433df2737d97223b371284e29683366f"
Jul 17 14:32:32.311055 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:32.311041    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq_odh-ai-gateway-infra(95c1fe0b-5527-4673-a2b8-aea4e3b2f126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126"
Jul 17 14:32:32.319571 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:32.319553    2575 scope.go:117] "RemoveContainer" containerID="e68233cb1c86d5bc4e406556b68c5446d4b35bc15778be73f22cec53ec30e55e"
Jul 17 14:32:33.318339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:33.318313    2575 scope.go:117] "RemoveContainer" containerID="9ff5912da41decfef611c115a0c18987433df2737d97223b371284e29683366f"
Jul 17 14:32:33.318706 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:33.318493    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq_odh-ai-gateway-infra(95c1fe0b-5527-4673-a2b8-aea4e3b2f126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126"
Jul 17 14:32:33.368853 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:33.368828    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"]
Jul 17 14:32:33.503745 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:33.503724    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"
Jul 17 14:32:33.607845 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:33.607784    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b76dd-dd62-4eaf-9600-bf9c34397b8e-maas-api-service-ca\") pod \"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e\" (UID: \"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e\") "
Jul 17 14:32:33.608111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:33.608092    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7b76dd-dd62-4eaf-9600-bf9c34397b8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" (UID: "3d7b76dd-dd62-4eaf-9600-bf9c34397b8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:32:33.709177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:33.709146    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b76dd-dd62-4eaf-9600-bf9c34397b8e-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:32:34.324210 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:34.324185    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"
Jul 17 14:32:34.324574 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:34.324183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5" event={"ID":"3d7b76dd-dd62-4eaf-9600-bf9c34397b8e","Type":"ContainerDied","Data":"82c86c98a0623f32c1ff8553e82c3154c1a5fa9f46f3ecf92a2e7eda7eb3a261"}
Jul 17 14:32:34.324574 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:34.324333    2575 scope.go:117] "RemoveContainer" containerID="11c96efe6eaf95fe5dc3d6ff157b0c9455587b332fdedeb31f6a9782439301a3"
Jul 17 14:32:34.358502 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:34.358476    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"]
Jul 17 14:32:34.360011 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:34.359991    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ngxr5"]
Jul 17 14:32:36.074838 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:36.074810    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" path="/var/lib/kubelet/pods/3d7b76dd-dd62-4eaf-9600-bf9c34397b8e/volumes"
Jul 17 14:32:41.839394 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839359    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"]
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839742    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839752    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839762    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839768    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839777    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839782    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839788    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839793    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba956fe-68fd-403e-a1a2-40b1f3911e22" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839846    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.839857 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.839857    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:41.844329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.844313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"
Jul 17 14:32:41.859108 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.859082    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"]
Jul 17 14:32:41.977533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:41.977505    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66c01d1-a15d-43a9-8e89-fe456261b76d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-dkbml\" (UID: \"d66c01d1-a15d-43a9-8e89-fe456261b76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"
Jul 17 14:32:42.078027 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:42.077998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66c01d1-a15d-43a9-8e89-fe456261b76d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-dkbml\" (UID: \"d66c01d1-a15d-43a9-8e89-fe456261b76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"
Jul 17 14:32:42.078588 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:42.078571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66c01d1-a15d-43a9-8e89-fe456261b76d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-dkbml\" (UID: \"d66c01d1-a15d-43a9-8e89-fe456261b76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"
Jul 17 14:32:42.154097 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:42.154032    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"
Jul 17 14:32:42.493841 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:42.493820    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"]
Jul 17 14:32:42.496233 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:32:42.496204    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66c01d1_a15d_43a9_8e89_fe456261b76d.slice/crio-8f53120f96557693e77dabb7c9b2ed23490a81e527d79c3b60ce9e766bf0142a WatchSource:0}: Error finding container 8f53120f96557693e77dabb7c9b2ed23490a81e527d79c3b60ce9e766bf0142a: Status 404 returned error can't find the container with id 8f53120f96557693e77dabb7c9b2ed23490a81e527d79c3b60ce9e766bf0142a
Jul 17 14:32:43.370988 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:43.370910    2575 generic.go:356] "Generic (PLEG): container finished" podID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerID="e9adb008117a3013137372b214ab1da1dab1717c5f51fa002297865238267eed" exitCode=6
Jul 17 14:32:43.371369 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:43.370998    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" event={"ID":"d66c01d1-a15d-43a9-8e89-fe456261b76d","Type":"ContainerDied","Data":"e9adb008117a3013137372b214ab1da1dab1717c5f51fa002297865238267eed"}
Jul 17 14:32:43.371369 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:43.371094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" event={"ID":"d66c01d1-a15d-43a9-8e89-fe456261b76d","Type":"ContainerStarted","Data":"8f53120f96557693e77dabb7c9b2ed23490a81e527d79c3b60ce9e766bf0142a"}
Jul 17 14:32:43.371369 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:43.371183    2575 scope.go:117] "RemoveContainer" containerID="e9adb008117a3013137372b214ab1da1dab1717c5f51fa002297865238267eed"
Jul 17 14:32:44.376826 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:44.376787    2575 generic.go:356] "Generic (PLEG): container finished" podID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerID="b8408d6fc8cc7b4e49cff5b5d6f99a26d8daca6bc1356e7862a9a99bfb0f4db7" exitCode=6
Jul 17 14:32:44.377266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:44.376845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" event={"ID":"d66c01d1-a15d-43a9-8e89-fe456261b76d","Type":"ContainerDied","Data":"b8408d6fc8cc7b4e49cff5b5d6f99a26d8daca6bc1356e7862a9a99bfb0f4db7"}
Jul 17 14:32:44.377266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:44.376898    2575 scope.go:117] "RemoveContainer" containerID="e9adb008117a3013137372b214ab1da1dab1717c5f51fa002297865238267eed"
Jul 17 14:32:44.377266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:44.377039    2575 scope.go:117] "RemoveContainer" containerID="b8408d6fc8cc7b4e49cff5b5d6f99a26d8daca6bc1356e7862a9a99bfb0f4db7"
Jul 17 14:32:44.377399 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:44.377307    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-dkbml_odh-ai-gateway-infra(d66c01d1-a15d-43a9-8e89-fe456261b76d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d"
Jul 17 14:32:45.383546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:45.383521    2575 scope.go:117] "RemoveContainer" containerID="b8408d6fc8cc7b4e49cff5b5d6f99a26d8daca6bc1356e7862a9a99bfb0f4db7"
Jul 17 14:32:45.383909 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:45.383682    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-dkbml_odh-ai-gateway-infra(d66c01d1-a15d-43a9-8e89-fe456261b76d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d"
Jul 17 14:32:47.069235 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:47.069206    2575 scope.go:117] "RemoveContainer" containerID="9ff5912da41decfef611c115a0c18987433df2737d97223b371284e29683366f"
Jul 17 14:32:47.394486 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:47.394455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" event={"ID":"95c1fe0b-5527-4673-a2b8-aea4e3b2f126","Type":"ContainerStarted","Data":"e4e5dda76ba03b44f239cbe02aeece945c088183496e04ad8c28bef075e78161"}
Jul 17 14:32:48.400750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:48.400714    2575 generic.go:356] "Generic (PLEG): container finished" podID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerID="e4e5dda76ba03b44f239cbe02aeece945c088183496e04ad8c28bef075e78161" exitCode=6
Jul 17 14:32:48.401130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:48.400757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" event={"ID":"95c1fe0b-5527-4673-a2b8-aea4e3b2f126","Type":"ContainerDied","Data":"e4e5dda76ba03b44f239cbe02aeece945c088183496e04ad8c28bef075e78161"}
Jul 17 14:32:48.401130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:48.400816    2575 scope.go:117] "RemoveContainer" containerID="9ff5912da41decfef611c115a0c18987433df2737d97223b371284e29683366f"
Jul 17 14:32:48.401130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:48.401068    2575 scope.go:117] "RemoveContainer" containerID="e4e5dda76ba03b44f239cbe02aeece945c088183496e04ad8c28bef075e78161"
Jul 17 14:32:48.401352 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:48.401253    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq_odh-ai-gateway-infra(95c1fe0b-5527-4673-a2b8-aea4e3b2f126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126"
Jul 17 14:32:48.423832 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:48.423798    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"]
Jul 17 14:32:49.533813 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:49.533786    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"
Jul 17 14:32:49.641111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:49.641080    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c1fe0b-5527-4673-a2b8-aea4e3b2f126-maas-api-service-ca\") pod \"95c1fe0b-5527-4673-a2b8-aea4e3b2f126\" (UID: \"95c1fe0b-5527-4673-a2b8-aea4e3b2f126\") "
Jul 17 14:32:49.641400 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:49.641381    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c1fe0b-5527-4673-a2b8-aea4e3b2f126-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95c1fe0b-5527-4673-a2b8-aea4e3b2f126" (UID: "95c1fe0b-5527-4673-a2b8-aea4e3b2f126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:32:49.742058 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:49.742031    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c1fe0b-5527-4673-a2b8-aea4e3b2f126-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:32:50.412424 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:50.412342    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"
Jul 17 14:32:50.412424 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:50.412372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq" event={"ID":"95c1fe0b-5527-4673-a2b8-aea4e3b2f126","Type":"ContainerDied","Data":"d6c276dede59e5297ce1f46c9276077a34e219f4acf7a4e38641e7bd1287dfad"}
Jul 17 14:32:50.412650 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:50.412429    2575 scope.go:117] "RemoveContainer" containerID="e4e5dda76ba03b44f239cbe02aeece945c088183496e04ad8c28bef075e78161"
Jul 17 14:32:50.430483 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:50.430457    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"]
Jul 17 14:32:50.433514 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:50.433492    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-zrxwq"]
Jul 17 14:32:52.077865 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.077829    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" path="/var/lib/kubelet/pods/95c1fe0b-5527-4673-a2b8-aea4e3b2f126/volumes"
Jul 17 14:32:52.569813 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.569773    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"]
Jul 17 14:32:52.570238 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570220    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.570370 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570240    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.570370 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570258    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.570370 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570266    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.570370 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570312    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.570370 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570322    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.570606 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570445    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7b76dd-dd62-4eaf-9600-bf9c34397b8e" containerName="revoke-keys"
Jul 17 14:32:52.570606 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570460    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.570606 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.570470    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:32:52.574958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.574937    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"
Jul 17 14:32:52.583009 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.582987    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"]
Jul 17 14:32:52.666804 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.666769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e88cedf-091c-4647-9d32-91fbe368b4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj\" (UID: \"5e88cedf-091c-4647-9d32-91fbe368b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"
Jul 17 14:32:52.767221 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.767192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e88cedf-091c-4647-9d32-91fbe368b4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj\" (UID: \"5e88cedf-091c-4647-9d32-91fbe368b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"
Jul 17 14:32:52.767918 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.767889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e88cedf-091c-4647-9d32-91fbe368b4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj\" (UID: \"5e88cedf-091c-4647-9d32-91fbe368b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"
Jul 17 14:32:52.886501 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:52.886436    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"
Jul 17 14:32:53.012336 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:53.012312    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"]
Jul 17 14:32:53.014625 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:32:53.014598    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e88cedf_091c_4647_9d32_91fbe368b4d3.slice/crio-4a82dc574adc63b17bfc3a6c5d7cf42a55ac7a475d2b98d5dfd7642ce88a95dd WatchSource:0}: Error finding container 4a82dc574adc63b17bfc3a6c5d7cf42a55ac7a475d2b98d5dfd7642ce88a95dd: Status 404 returned error can't find the container with id 4a82dc574adc63b17bfc3a6c5d7cf42a55ac7a475d2b98d5dfd7642ce88a95dd
Jul 17 14:32:53.429985 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:53.429953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" event={"ID":"5e88cedf-091c-4647-9d32-91fbe368b4d3","Type":"ContainerStarted","Data":"93e64b283a6b90f359a37a9811fc18c3933807cea36e1c0e77ec4522fffd13fa"}
Jul 17 14:32:53.430373 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:53.430006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" event={"ID":"5e88cedf-091c-4647-9d32-91fbe368b4d3","Type":"ContainerStarted","Data":"4a82dc574adc63b17bfc3a6c5d7cf42a55ac7a475d2b98d5dfd7642ce88a95dd"}
Jul 17 14:32:53.449565 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:53.448803    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" podStartSLOduration=1.448786229 podStartE2EDuration="1.448786229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:32:53.44603101 +0000 UTC m=+2851.833378444" watchObservedRunningTime="2026-07-17 14:32:53.448786229 +0000 UTC m=+2851.836133670"
Jul 17 14:32:54.436260 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:54.436223    2575 generic.go:356] "Generic (PLEG): container finished" podID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerID="93e64b283a6b90f359a37a9811fc18c3933807cea36e1c0e77ec4522fffd13fa" exitCode=6
Jul 17 14:32:54.436699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:54.436313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" event={"ID":"5e88cedf-091c-4647-9d32-91fbe368b4d3","Type":"ContainerDied","Data":"93e64b283a6b90f359a37a9811fc18c3933807cea36e1c0e77ec4522fffd13fa"}
Jul 17 14:32:54.436699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:54.436435    2575 scope.go:117] "RemoveContainer" containerID="93e64b283a6b90f359a37a9811fc18c3933807cea36e1c0e77ec4522fffd13fa"
Jul 17 14:32:55.443841 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:55.443751    2575 generic.go:356] "Generic (PLEG): container finished" podID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerID="88be0a69991730d1665841a4de75b2f71ce2b729f7ebc57679a65ac12468bc8a" exitCode=6
Jul 17 14:32:55.444255 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:55.443839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" event={"ID":"5e88cedf-091c-4647-9d32-91fbe368b4d3","Type":"ContainerDied","Data":"88be0a69991730d1665841a4de75b2f71ce2b729f7ebc57679a65ac12468bc8a"}
Jul 17 14:32:55.444255 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:55.443942    2575 scope.go:117] "RemoveContainer" containerID="93e64b283a6b90f359a37a9811fc18c3933807cea36e1c0e77ec4522fffd13fa"
Jul 17 14:32:55.444255 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:55.444151    2575 scope.go:117] "RemoveContainer" containerID="88be0a69991730d1665841a4de75b2f71ce2b729f7ebc57679a65ac12468bc8a"
Jul 17 14:32:55.444421 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:55.444362    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj_odh-ai-gateway-infra(5e88cedf-091c-4647-9d32-91fbe368b4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3"
Jul 17 14:32:56.450454 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:32:56.450425    2575 scope.go:117] "RemoveContainer" containerID="88be0a69991730d1665841a4de75b2f71ce2b729f7ebc57679a65ac12468bc8a"
Jul 17 14:32:56.450831 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:32:56.450608    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj_odh-ai-gateway-infra(5e88cedf-091c-4647-9d32-91fbe368b4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3"
Jul 17 14:33:00.069959 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:00.069933    2575 scope.go:117] "RemoveContainer" containerID="b8408d6fc8cc7b4e49cff5b5d6f99a26d8daca6bc1356e7862a9a99bfb0f4db7"
Jul 17 14:33:00.471559 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:00.471532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" event={"ID":"d66c01d1-a15d-43a9-8e89-fe456261b76d","Type":"ContainerStarted","Data":"974ee939c43f10118be88ae2d49be6af2d86932c4b4103e95a606ae2ccaad60a"}
Jul 17 14:33:00.491897 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:00.491855    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" podStartSLOduration=19.491841778 podStartE2EDuration="19.491841778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:33:00.489391371 +0000 UTC m=+2858.876738810" watchObservedRunningTime="2026-07-17 14:33:00.491841778 +0000 UTC m=+2858.879189217"
Jul 17 14:33:01.479521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:01.479486    2575 generic.go:356] "Generic (PLEG): container finished" podID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerID="974ee939c43f10118be88ae2d49be6af2d86932c4b4103e95a606ae2ccaad60a" exitCode=6
Jul 17 14:33:01.479895 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:01.479527    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" event={"ID":"d66c01d1-a15d-43a9-8e89-fe456261b76d","Type":"ContainerDied","Data":"974ee939c43f10118be88ae2d49be6af2d86932c4b4103e95a606ae2ccaad60a"}
Jul 17 14:33:01.479895 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:01.479572    2575 scope.go:117] "RemoveContainer" containerID="b8408d6fc8cc7b4e49cff5b5d6f99a26d8daca6bc1356e7862a9a99bfb0f4db7"
Jul 17 14:33:01.479895 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:01.479807    2575 scope.go:117] "RemoveContainer" containerID="974ee939c43f10118be88ae2d49be6af2d86932c4b4103e95a606ae2ccaad60a"
Jul 17 14:33:01.480052 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:01.480026    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-dkbml_odh-ai-gateway-infra(d66c01d1-a15d-43a9-8e89-fe456261b76d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d"
Jul 17 14:33:01.509543 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:01.509523    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"]
Jul 17 14:33:02.616949 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:02.616925    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"
Jul 17 14:33:02.749374 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:02.749309    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66c01d1-a15d-43a9-8e89-fe456261b76d-maas-api-service-ca\") pod \"d66c01d1-a15d-43a9-8e89-fe456261b76d\" (UID: \"d66c01d1-a15d-43a9-8e89-fe456261b76d\") "
Jul 17 14:33:02.749666 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:02.749643    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66c01d1-a15d-43a9-8e89-fe456261b76d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d66c01d1-a15d-43a9-8e89-fe456261b76d" (UID: "d66c01d1-a15d-43a9-8e89-fe456261b76d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:33:02.850209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:02.850165    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66c01d1-a15d-43a9-8e89-fe456261b76d-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:33:03.492002 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:03.491969    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"
Jul 17 14:33:03.492002 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:03.491987    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml" event={"ID":"d66c01d1-a15d-43a9-8e89-fe456261b76d","Type":"ContainerDied","Data":"8f53120f96557693e77dabb7c9b2ed23490a81e527d79c3b60ce9e766bf0142a"}
Jul 17 14:33:03.492233 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:03.492047    2575 scope.go:117] "RemoveContainer" containerID="974ee939c43f10118be88ae2d49be6af2d86932c4b4103e95a606ae2ccaad60a"
Jul 17 14:33:03.514642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:03.514612    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"]
Jul 17 14:33:03.523759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:03.523737    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-dkbml"]
Jul 17 14:33:04.074835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:04.074804    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" path="/var/lib/kubelet/pods/d66c01d1-a15d-43a9-8e89-fe456261b76d/volumes"
Jul 17 14:33:12.072474 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:12.072450    2575 scope.go:117] "RemoveContainer" containerID="88be0a69991730d1665841a4de75b2f71ce2b729f7ebc57679a65ac12468bc8a"
Jul 17 14:33:12.537641 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:12.537611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" event={"ID":"5e88cedf-091c-4647-9d32-91fbe368b4d3","Type":"ContainerStarted","Data":"24eb10a363681cf500183b365da4d3bf621fa5be974664c4fdcfb5aa4b47ada1"}
Jul 17 14:33:13.409777 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.409743    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"]
Jul 17 14:33:13.410146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410102    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410113    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410133    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410139    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410146 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410146    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410338 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410152    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410338 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410209    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410338 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410218    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:13.410338 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.410227    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c1fe0b-5527-4673-a2b8-aea4e3b2f126" containerName="revoke-keys"
Jul 17 14:33:13.413307 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.413271    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"
Jul 17 14:33:13.423198 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.423175    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"]
Jul 17 14:33:13.434074 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.434044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785aaaf3-2618-4fa1-8d2a-0137b3c04b78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-txkxd\" (UID: \"785aaaf3-2618-4fa1-8d2a-0137b3c04b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"
Jul 17 14:33:13.534919 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.534885    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785aaaf3-2618-4fa1-8d2a-0137b3c04b78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-txkxd\" (UID: \"785aaaf3-2618-4fa1-8d2a-0137b3c04b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"
Jul 17 14:33:13.535513 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.535495    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785aaaf3-2618-4fa1-8d2a-0137b3c04b78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-txkxd\" (UID: \"785aaaf3-2618-4fa1-8d2a-0137b3c04b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"
Jul 17 14:33:13.543595 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.543570    2575 generic.go:356] "Generic (PLEG): container finished" podID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerID="24eb10a363681cf500183b365da4d3bf621fa5be974664c4fdcfb5aa4b47ada1" exitCode=6
Jul 17 14:33:13.543692 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.543645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" event={"ID":"5e88cedf-091c-4647-9d32-91fbe368b4d3","Type":"ContainerDied","Data":"24eb10a363681cf500183b365da4d3bf621fa5be974664c4fdcfb5aa4b47ada1"}
Jul 17 14:33:13.543733 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.543695    2575 scope.go:117] "RemoveContainer" containerID="88be0a69991730d1665841a4de75b2f71ce2b729f7ebc57679a65ac12468bc8a"
Jul 17 14:33:13.543975 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.543960    2575 scope.go:117] "RemoveContainer" containerID="24eb10a363681cf500183b365da4d3bf621fa5be974664c4fdcfb5aa4b47ada1"
Jul 17 14:33:13.544158 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:13.544136    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj_odh-ai-gateway-infra(5e88cedf-091c-4647-9d32-91fbe368b4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3"
Jul 17 14:33:13.574900 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.574877    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"]
Jul 17 14:33:13.724085 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.724062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"
Jul 17 14:33:13.861267 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:13.861243    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"]
Jul 17 14:33:13.863359 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:33:13.863325    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785aaaf3_2618_4fa1_8d2a_0137b3c04b78.slice/crio-a52e1ed275e77d24418c1a5eb96c6d0ae385b7824fbf02a50bde225404faa64e WatchSource:0}: Error finding container a52e1ed275e77d24418c1a5eb96c6d0ae385b7824fbf02a50bde225404faa64e: Status 404 returned error can't find the container with id a52e1ed275e77d24418c1a5eb96c6d0ae385b7824fbf02a50bde225404faa64e
Jul 17 14:33:14.551468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.551426    2575 generic.go:356] "Generic (PLEG): container finished" podID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerID="0b9000ccc3df0a96c1805a1202cdf3b3bba74b9da0b00374c8489a9ffa24bd5e" exitCode=6
Jul 17 14:33:14.551831 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.551505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" event={"ID":"785aaaf3-2618-4fa1-8d2a-0137b3c04b78","Type":"ContainerDied","Data":"0b9000ccc3df0a96c1805a1202cdf3b3bba74b9da0b00374c8489a9ffa24bd5e"}
Jul 17 14:33:14.551831 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.551554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" event={"ID":"785aaaf3-2618-4fa1-8d2a-0137b3c04b78","Type":"ContainerStarted","Data":"a52e1ed275e77d24418c1a5eb96c6d0ae385b7824fbf02a50bde225404faa64e"}
Jul 17 14:33:14.551831 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.551629    2575 scope.go:117] "RemoveContainer" containerID="0b9000ccc3df0a96c1805a1202cdf3b3bba74b9da0b00374c8489a9ffa24bd5e"
Jul 17 14:33:14.671623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.671602    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"
Jul 17 14:33:14.746000 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.745977    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e88cedf-091c-4647-9d32-91fbe368b4d3-maas-api-service-ca\") pod \"5e88cedf-091c-4647-9d32-91fbe368b4d3\" (UID: \"5e88cedf-091c-4647-9d32-91fbe368b4d3\") "
Jul 17 14:33:14.746352 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.746332    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e88cedf-091c-4647-9d32-91fbe368b4d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e88cedf-091c-4647-9d32-91fbe368b4d3" (UID: "5e88cedf-091c-4647-9d32-91fbe368b4d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:33:14.847375 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:14.847353    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e88cedf-091c-4647-9d32-91fbe368b4d3-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:33:15.557229 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.557160    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"
Jul 17 14:33:15.557229 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.557186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj" event={"ID":"5e88cedf-091c-4647-9d32-91fbe368b4d3","Type":"ContainerDied","Data":"4a82dc574adc63b17bfc3a6c5d7cf42a55ac7a475d2b98d5dfd7642ce88a95dd"}
Jul 17 14:33:15.557763 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.557243    2575 scope.go:117] "RemoveContainer" containerID="24eb10a363681cf500183b365da4d3bf621fa5be974664c4fdcfb5aa4b47ada1"
Jul 17 14:33:15.559232 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.559206    2575 generic.go:356] "Generic (PLEG): container finished" podID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerID="027a5c64ba1062de10c5dc50cd55bed931a531d78a05905a9f68e2eae7d20a01" exitCode=6
Jul 17 14:33:15.559357 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.559292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" event={"ID":"785aaaf3-2618-4fa1-8d2a-0137b3c04b78","Type":"ContainerDied","Data":"027a5c64ba1062de10c5dc50cd55bed931a531d78a05905a9f68e2eae7d20a01"}
Jul 17 14:33:15.561025 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.561003    2575 scope.go:117] "RemoveContainer" containerID="027a5c64ba1062de10c5dc50cd55bed931a531d78a05905a9f68e2eae7d20a01"
Jul 17 14:33:15.561258 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:15.561232    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-txkxd_odh-ai-gateway-infra(785aaaf3-2618-4fa1-8d2a-0137b3c04b78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78"
Jul 17 14:33:15.566412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.566399    2575 scope.go:117] "RemoveContainer" containerID="0b9000ccc3df0a96c1805a1202cdf3b3bba74b9da0b00374c8489a9ffa24bd5e"
Jul 17 14:33:15.591663 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.591619    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"]
Jul 17 14:33:15.593316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:15.593297    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-qsdwj"]
Jul 17 14:33:16.074173 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:16.074143    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" path="/var/lib/kubelet/pods/5e88cedf-091c-4647-9d32-91fbe368b4d3/volumes"
Jul 17 14:33:16.571930 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:16.571906    2575 scope.go:117] "RemoveContainer" containerID="027a5c64ba1062de10c5dc50cd55bed931a531d78a05905a9f68e2eae7d20a01"
Jul 17 14:33:16.572259 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:16.572055    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-txkxd_odh-ai-gateway-infra(785aaaf3-2618-4fa1-8d2a-0137b3c04b78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78"
Jul 17 14:33:29.069333 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.069303    2575 scope.go:117] "RemoveContainer" containerID="027a5c64ba1062de10c5dc50cd55bed931a531d78a05905a9f68e2eae7d20a01"
Jul 17 14:33:29.240723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.240699    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"]
Jul 17 14:33:29.241101 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.241088    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:29.241158 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.241106    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:29.241158 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.241113    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:29.241158 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.241119    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:29.241248 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.241184    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:29.241248 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.241193    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66c01d1-a15d-43a9-8e89-fe456261b76d" containerName="revoke-keys"
Jul 17 14:33:29.241248 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.241200    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:29.245511 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.245497    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"
Jul 17 14:33:29.257996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.257969    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"]
Jul 17 14:33:29.361701 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.361640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz\" (UID: \"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"
Jul 17 14:33:29.462351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.462323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz\" (UID: \"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"
Jul 17 14:33:29.462877 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.462858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz\" (UID: \"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"
Jul 17 14:33:29.554930 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.554902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"
Jul 17 14:33:29.633297 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.633214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" event={"ID":"785aaaf3-2618-4fa1-8d2a-0137b3c04b78","Type":"ContainerStarted","Data":"17e67f7c6f9f3e52623f3c38c1df27e9608dbd87bc0187a1a312793695aa423a"}
Jul 17 14:33:29.652586 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.652540    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" podStartSLOduration=16.652520773 podStartE2EDuration="16.652520773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:33:29.651557749 +0000 UTC m=+2888.038905189" watchObservedRunningTime="2026-07-17 14:33:29.652520773 +0000 UTC m=+2888.039868213"
Jul 17 14:33:29.683403 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:29.683375    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"]
Jul 17 14:33:29.726186 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:33:29.726154    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba6af8e_b5c2_4f6d_8bfa_a5f66ad78c59.slice/crio-28245898c415395d3793753d6b44aab43ff7ce021cc16973455a092dbe0ce313 WatchSource:0}: Error finding container 28245898c415395d3793753d6b44aab43ff7ce021cc16973455a092dbe0ce313: Status 404 returned error can't find the container with id 28245898c415395d3793753d6b44aab43ff7ce021cc16973455a092dbe0ce313
Jul 17 14:33:30.639393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.639331    2575 generic.go:356] "Generic (PLEG): container finished" podID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerID="17e67f7c6f9f3e52623f3c38c1df27e9608dbd87bc0187a1a312793695aa423a" exitCode=6
Jul 17 14:33:30.639748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.639397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" event={"ID":"785aaaf3-2618-4fa1-8d2a-0137b3c04b78","Type":"ContainerDied","Data":"17e67f7c6f9f3e52623f3c38c1df27e9608dbd87bc0187a1a312793695aa423a"}
Jul 17 14:33:30.639748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.639458    2575 scope.go:117] "RemoveContainer" containerID="027a5c64ba1062de10c5dc50cd55bed931a531d78a05905a9f68e2eae7d20a01"
Jul 17 14:33:30.639748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.639709    2575 scope.go:117] "RemoveContainer" containerID="17e67f7c6f9f3e52623f3c38c1df27e9608dbd87bc0187a1a312793695aa423a"
Jul 17 14:33:30.639925 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:30.639905    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-txkxd_odh-ai-gateway-infra(785aaaf3-2618-4fa1-8d2a-0137b3c04b78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78"
Jul 17 14:33:30.640956 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.640940    2575 generic.go:356] "Generic (PLEG): container finished" podID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerID="cf558d81fa33fcb9ddd0cbb1384bfcfde40218f5bca3cb503d3d6524a0df9083" exitCode=6
Jul 17 14:33:30.641035 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.640966    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" event={"ID":"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59","Type":"ContainerDied","Data":"cf558d81fa33fcb9ddd0cbb1384bfcfde40218f5bca3cb503d3d6524a0df9083"}
Jul 17 14:33:30.641035 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.640994    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" event={"ID":"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59","Type":"ContainerStarted","Data":"28245898c415395d3793753d6b44aab43ff7ce021cc16973455a092dbe0ce313"}
Jul 17 14:33:30.641170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.641155    2575 scope.go:117] "RemoveContainer" containerID="cf558d81fa33fcb9ddd0cbb1384bfcfde40218f5bca3cb503d3d6524a0df9083"
Jul 17 14:33:30.676633 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:30.676607    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"]
Jul 17 14:33:31.649945 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.649919    2575 generic.go:356] "Generic (PLEG): container finished" podID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerID="80a11b7da0f795286ce09944e5c4ce532789943504049f5dfb972abd3dcee455" exitCode=6
Jul 17 14:33:31.650308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.650010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" event={"ID":"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59","Type":"ContainerDied","Data":"80a11b7da0f795286ce09944e5c4ce532789943504049f5dfb972abd3dcee455"}
Jul 17 14:33:31.650308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.650084    2575 scope.go:117] "RemoveContainer" containerID="cf558d81fa33fcb9ddd0cbb1384bfcfde40218f5bca3cb503d3d6524a0df9083"
Jul 17 14:33:31.650308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.650138    2575 scope.go:117] "RemoveContainer" containerID="80a11b7da0f795286ce09944e5c4ce532789943504049f5dfb972abd3dcee455"
Jul 17 14:33:31.650459 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:31.650344    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz_odh-ai-gateway-infra(aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59"
Jul 17 14:33:31.780112 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.780087    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"
Jul 17 14:33:31.883768 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.883740    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785aaaf3-2618-4fa1-8d2a-0137b3c04b78-maas-api-service-ca\") pod \"785aaaf3-2618-4fa1-8d2a-0137b3c04b78\" (UID: \"785aaaf3-2618-4fa1-8d2a-0137b3c04b78\") "
Jul 17 14:33:31.884049 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.884030    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785aaaf3-2618-4fa1-8d2a-0137b3c04b78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "785aaaf3-2618-4fa1-8d2a-0137b3c04b78" (UID: "785aaaf3-2618-4fa1-8d2a-0137b3c04b78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:33:31.984931 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:31.984908    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785aaaf3-2618-4fa1-8d2a-0137b3c04b78-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:33:32.655136 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:32.655111    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"
Jul 17 14:33:32.655136 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:32.655120    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd" event={"ID":"785aaaf3-2618-4fa1-8d2a-0137b3c04b78","Type":"ContainerDied","Data":"a52e1ed275e77d24418c1a5eb96c6d0ae385b7824fbf02a50bde225404faa64e"}
Jul 17 14:33:32.655605 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:32.655173    2575 scope.go:117] "RemoveContainer" containerID="17e67f7c6f9f3e52623f3c38c1df27e9608dbd87bc0187a1a312793695aa423a"
Jul 17 14:33:32.657195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:32.657175    2575 scope.go:117] "RemoveContainer" containerID="80a11b7da0f795286ce09944e5c4ce532789943504049f5dfb972abd3dcee455"
Jul 17 14:33:32.657585 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:32.657559    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz_odh-ai-gateway-infra(aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59"
Jul 17 14:33:32.722101 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:32.722077    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"]
Jul 17 14:33:32.727272 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:32.727245    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-txkxd"]
Jul 17 14:33:34.074686 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:34.074657    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" path="/var/lib/kubelet/pods/785aaaf3-2618-4fa1-8d2a-0137b3c04b78/volumes"
Jul 17 14:33:41.889668 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.889634    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"]
Jul 17 14:33:41.890068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890000    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:41.890068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890011    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:41.890068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890023    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:41.890068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890030    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:41.890068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890038    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:41.890068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890044    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:41.890303 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890108    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:41.890303 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890115    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:41.890303 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.890123    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e88cedf-091c-4647-9d32-91fbe368b4d3" containerName="revoke-keys"
Jul 17 14:33:41.894300 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.894265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"
Jul 17 14:33:41.902392 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.902368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"]
Jul 17 14:33:41.962919 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:41.962893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61522f3e-92dd-4a4b-99e0-7fbd26bf5017-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-rhbmt\" (UID: \"61522f3e-92dd-4a4b-99e0-7fbd26bf5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"
Jul 17 14:33:42.063807 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:42.063767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61522f3e-92dd-4a4b-99e0-7fbd26bf5017-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-rhbmt\" (UID: \"61522f3e-92dd-4a4b-99e0-7fbd26bf5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"
Jul 17 14:33:42.064344 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:42.064325    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61522f3e-92dd-4a4b-99e0-7fbd26bf5017-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-rhbmt\" (UID: \"61522f3e-92dd-4a4b-99e0-7fbd26bf5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"
Jul 17 14:33:42.204979 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:42.204917    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"
Jul 17 14:33:42.355861 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:33:42.355831    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61522f3e_92dd_4a4b_99e0_7fbd26bf5017.slice/crio-e77cc71cd3e8fccc13360d52b1e7d26213a9de8a9d660f497946f8687bcc6e0c WatchSource:0}: Error finding container e77cc71cd3e8fccc13360d52b1e7d26213a9de8a9d660f497946f8687bcc6e0c: Status 404 returned error can't find the container with id e77cc71cd3e8fccc13360d52b1e7d26213a9de8a9d660f497946f8687bcc6e0c
Jul 17 14:33:42.356140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:42.356118    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"]
Jul 17 14:33:42.702648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:42.702622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" event={"ID":"61522f3e-92dd-4a4b-99e0-7fbd26bf5017","Type":"ContainerStarted","Data":"8c886081f18eef3902caf21b90a5f8d04ee00f2cb6e4f63c4a759b28a48a0b95"}
Jul 17 14:33:42.702754 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:42.702672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" event={"ID":"61522f3e-92dd-4a4b-99e0-7fbd26bf5017","Type":"ContainerStarted","Data":"e77cc71cd3e8fccc13360d52b1e7d26213a9de8a9d660f497946f8687bcc6e0c"}
Jul 17 14:33:42.722744 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:42.722701    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" podStartSLOduration=1.722687658 podStartE2EDuration="1.722687658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:33:42.721291725 +0000 UTC m=+2901.108639155" watchObservedRunningTime="2026-07-17 14:33:42.722687658 +0000 UTC m=+2901.110035097"
Jul 17 14:33:43.708591 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:43.708553    2575 generic.go:356] "Generic (PLEG): container finished" podID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerID="8c886081f18eef3902caf21b90a5f8d04ee00f2cb6e4f63c4a759b28a48a0b95" exitCode=6
Jul 17 14:33:43.708991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:43.708630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" event={"ID":"61522f3e-92dd-4a4b-99e0-7fbd26bf5017","Type":"ContainerDied","Data":"8c886081f18eef3902caf21b90a5f8d04ee00f2cb6e4f63c4a759b28a48a0b95"}
Jul 17 14:33:43.708991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:43.708853    2575 scope.go:117] "RemoveContainer" containerID="8c886081f18eef3902caf21b90a5f8d04ee00f2cb6e4f63c4a759b28a48a0b95"
Jul 17 14:33:44.715075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:44.715045    2575 generic.go:356] "Generic (PLEG): container finished" podID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerID="5a3bc53edfdf8badebbb901ce0fe238efb2b4e40758da311cb4082eff5ecb258" exitCode=6
Jul 17 14:33:44.715545 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:44.715110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" event={"ID":"61522f3e-92dd-4a4b-99e0-7fbd26bf5017","Type":"ContainerDied","Data":"5a3bc53edfdf8badebbb901ce0fe238efb2b4e40758da311cb4082eff5ecb258"}
Jul 17 14:33:44.715545 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:44.715158    2575 scope.go:117] "RemoveContainer" containerID="8c886081f18eef3902caf21b90a5f8d04ee00f2cb6e4f63c4a759b28a48a0b95"
Jul 17 14:33:44.715545 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:44.715386    2575 scope.go:117] "RemoveContainer" containerID="5a3bc53edfdf8badebbb901ce0fe238efb2b4e40758da311cb4082eff5ecb258"
Jul 17 14:33:44.715701 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:44.715638    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-rhbmt_odh-ai-gateway-infra(61522f3e-92dd-4a4b-99e0-7fbd26bf5017)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017"
Jul 17 14:33:45.721442 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:45.721416    2575 scope.go:117] "RemoveContainer" containerID="5a3bc53edfdf8badebbb901ce0fe238efb2b4e40758da311cb4082eff5ecb258"
Jul 17 14:33:45.721809 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:45.721589    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-rhbmt_odh-ai-gateway-infra(61522f3e-92dd-4a4b-99e0-7fbd26bf5017)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017"
Jul 17 14:33:46.069481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:46.069401    2575 scope.go:117] "RemoveContainer" containerID="80a11b7da0f795286ce09944e5c4ce532789943504049f5dfb972abd3dcee455"
Jul 17 14:33:46.727047 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:46.727021    2575 generic.go:356] "Generic (PLEG): container finished" podID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerID="1145778184f6e050e74c11e141802d2b6ef9ba5b3566a4cf641d2b0df4454b96" exitCode=6
Jul 17 14:33:46.727367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:46.727081    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" event={"ID":"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59","Type":"ContainerDied","Data":"1145778184f6e050e74c11e141802d2b6ef9ba5b3566a4cf641d2b0df4454b96"}
Jul 17 14:33:46.727367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:46.727126    2575 scope.go:117] "RemoveContainer" containerID="80a11b7da0f795286ce09944e5c4ce532789943504049f5dfb972abd3dcee455"
Jul 17 14:33:46.727457 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:46.727380    2575 scope.go:117] "RemoveContainer" containerID="1145778184f6e050e74c11e141802d2b6ef9ba5b3566a4cf641d2b0df4454b96"
Jul 17 14:33:46.727570 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:46.727551    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz_odh-ai-gateway-infra(aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59"
Jul 17 14:33:47.765671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:47.765640    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"]
Jul 17 14:33:47.896226 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:47.896205    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"
Jul 17 14:33:48.015861 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.015801    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59-maas-api-service-ca\") pod \"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59\" (UID: \"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59\") "
Jul 17 14:33:48.016156 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.016137    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" (UID: "aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:33:48.116396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.116376    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:33:48.746126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.746098    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"
Jul 17 14:33:48.746126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.746108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz" event={"ID":"aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59","Type":"ContainerDied","Data":"28245898c415395d3793753d6b44aab43ff7ce021cc16973455a092dbe0ce313"}
Jul 17 14:33:48.746354 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.746196    2575 scope.go:117] "RemoveContainer" containerID="1145778184f6e050e74c11e141802d2b6ef9ba5b3566a4cf641d2b0df4454b96"
Jul 17 14:33:48.764927 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.764902    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"]
Jul 17 14:33:48.780053 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:48.780029    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-5bdxz"]
Jul 17 14:33:50.074620 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:50.074586    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" path="/var/lib/kubelet/pods/aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59/volumes"
Jul 17 14:33:52.610553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610524    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"]
Jul 17 14:33:52.610940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610926    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:52.610940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610941    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:52.611031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610953    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610959    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610966    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610971    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610982    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.610987    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611199 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.611046    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611199 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.611054    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:33:52.611199 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.611062    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="785aaaf3-2618-4fa1-8d2a-0137b3c04b78" containerName="revoke-keys"
Jul 17 14:33:52.615329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.615313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"
Jul 17 14:33:52.622593 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.622573    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"]
Jul 17 14:33:52.753522 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.753500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0194f7-5cda-48a0-bd40-8631db137ecc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z\" (UID: \"2e0194f7-5cda-48a0-bd40-8631db137ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"
Jul 17 14:33:52.854138 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.854112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0194f7-5cda-48a0-bd40-8631db137ecc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z\" (UID: \"2e0194f7-5cda-48a0-bd40-8631db137ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"
Jul 17 14:33:52.854689 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.854673    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0194f7-5cda-48a0-bd40-8631db137ecc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z\" (UID: \"2e0194f7-5cda-48a0-bd40-8631db137ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"
Jul 17 14:33:52.925781 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:52.925715    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"
Jul 17 14:33:53.054125 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:53.054102    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"]
Jul 17 14:33:53.056041 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:33:53.056013    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0194f7_5cda_48a0_bd40_8631db137ecc.slice/crio-1d7c5b0f4e43d92c7ef6654863efc8d8a32e53bdafd8c0cf4ff8e46e72eeaa8c WatchSource:0}: Error finding container 1d7c5b0f4e43d92c7ef6654863efc8d8a32e53bdafd8c0cf4ff8e46e72eeaa8c: Status 404 returned error can't find the container with id 1d7c5b0f4e43d92c7ef6654863efc8d8a32e53bdafd8c0cf4ff8e46e72eeaa8c
Jul 17 14:33:53.776972 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:53.776945    2575 generic.go:356] "Generic (PLEG): container finished" podID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerID="591a6081663348a5f06ebe418b9e4eb2e08b33b2395d6c86f972c1f94439ded2" exitCode=6
Jul 17 14:33:53.777330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:53.777036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" event={"ID":"2e0194f7-5cda-48a0-bd40-8631db137ecc","Type":"ContainerDied","Data":"591a6081663348a5f06ebe418b9e4eb2e08b33b2395d6c86f972c1f94439ded2"}
Jul 17 14:33:53.777330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:53.777092    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" event={"ID":"2e0194f7-5cda-48a0-bd40-8631db137ecc","Type":"ContainerStarted","Data":"1d7c5b0f4e43d92c7ef6654863efc8d8a32e53bdafd8c0cf4ff8e46e72eeaa8c"}
Jul 17 14:33:53.777330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:53.777154    2575 scope.go:117] "RemoveContainer" containerID="591a6081663348a5f06ebe418b9e4eb2e08b33b2395d6c86f972c1f94439ded2"
Jul 17 14:33:54.783482 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:54.783446    2575 generic.go:356] "Generic (PLEG): container finished" podID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerID="455eb2079ee1b24412ea29858ab3e45337e71936c92e581a48bcf03d8dd39423" exitCode=6
Jul 17 14:33:54.783873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:54.783510    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" event={"ID":"2e0194f7-5cda-48a0-bd40-8631db137ecc","Type":"ContainerDied","Data":"455eb2079ee1b24412ea29858ab3e45337e71936c92e581a48bcf03d8dd39423"}
Jul 17 14:33:54.783873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:54.783576    2575 scope.go:117] "RemoveContainer" containerID="591a6081663348a5f06ebe418b9e4eb2e08b33b2395d6c86f972c1f94439ded2"
Jul 17 14:33:54.783873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:54.783765    2575 scope.go:117] "RemoveContainer" containerID="455eb2079ee1b24412ea29858ab3e45337e71936c92e581a48bcf03d8dd39423"
Jul 17 14:33:54.784066 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:54.783980    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z_odh-ai-gateway-infra(2e0194f7-5cda-48a0-bd40-8631db137ecc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc"
Jul 17 14:33:55.790243 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:55.790214    2575 scope.go:117] "RemoveContainer" containerID="455eb2079ee1b24412ea29858ab3e45337e71936c92e581a48bcf03d8dd39423"
Jul 17 14:33:55.790637 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:55.790436    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z_odh-ai-gateway-infra(2e0194f7-5cda-48a0-bd40-8631db137ecc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc"
Jul 17 14:33:59.069608 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:59.069579    2575 scope.go:117] "RemoveContainer" containerID="5a3bc53edfdf8badebbb901ce0fe238efb2b4e40758da311cb4082eff5ecb258"
Jul 17 14:33:59.811505 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:59.811478    2575 generic.go:356] "Generic (PLEG): container finished" podID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerID="d9980fd903401517b935c11beddb209bc8990dccd28d3d0f0da9867b1379f30f" exitCode=6
Jul 17 14:33:59.811679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:59.811549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" event={"ID":"61522f3e-92dd-4a4b-99e0-7fbd26bf5017","Type":"ContainerDied","Data":"d9980fd903401517b935c11beddb209bc8990dccd28d3d0f0da9867b1379f30f"}
Jul 17 14:33:59.811679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:59.811604    2575 scope.go:117] "RemoveContainer" containerID="5a3bc53edfdf8badebbb901ce0fe238efb2b4e40758da311cb4082eff5ecb258"
Jul 17 14:33:59.811853 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:33:59.811829    2575 scope.go:117] "RemoveContainer" containerID="d9980fd903401517b935c11beddb209bc8990dccd28d3d0f0da9867b1379f30f"
Jul 17 14:33:59.812116 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:33:59.812031    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-rhbmt_odh-ai-gateway-infra(61522f3e-92dd-4a4b-99e0-7fbd26bf5017)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017"
Jul 17 14:34:00.855242 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:00.855204    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"]
Jul 17 14:34:00.992931 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:00.992900    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"
Jul 17 14:34:01.116750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.116675    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61522f3e-92dd-4a4b-99e0-7fbd26bf5017-maas-api-service-ca\") pod \"61522f3e-92dd-4a4b-99e0-7fbd26bf5017\" (UID: \"61522f3e-92dd-4a4b-99e0-7fbd26bf5017\") "
Jul 17 14:34:01.117013 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.116992    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61522f3e-92dd-4a4b-99e0-7fbd26bf5017-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61522f3e-92dd-4a4b-99e0-7fbd26bf5017" (UID: "61522f3e-92dd-4a4b-99e0-7fbd26bf5017"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:34:01.217768 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.217744    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61522f3e-92dd-4a4b-99e0-7fbd26bf5017-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:34:01.825479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.825435    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt" event={"ID":"61522f3e-92dd-4a4b-99e0-7fbd26bf5017","Type":"ContainerDied","Data":"e77cc71cd3e8fccc13360d52b1e7d26213a9de8a9d660f497946f8687bcc6e0c"}
Jul 17 14:34:01.825479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.825471    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"
Jul 17 14:34:01.825707 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.825518    2575 scope.go:117] "RemoveContainer" containerID="d9980fd903401517b935c11beddb209bc8990dccd28d3d0f0da9867b1379f30f"
Jul 17 14:34:01.854482 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.854454    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"]
Jul 17 14:34:01.861766 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:01.861743    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-rhbmt"]
Jul 17 14:34:02.075638 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:02.075546    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" path="/var/lib/kubelet/pods/61522f3e-92dd-4a4b-99e0-7fbd26bf5017/volumes"
Jul 17 14:34:03.455552 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455522    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"]
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455878    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455889    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455903    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455909    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455919    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455925    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455984    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.455990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455991    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.456242 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.455997    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61522f3e-92dd-4a4b-99e0-7fbd26bf5017" containerName="revoke-keys"
Jul 17 14:34:03.456242 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.456003    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba6af8e-b5c2-4f6d-8bfa-a5f66ad78c59" containerName="revoke-keys"
Jul 17 14:34:03.458783 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.458768    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"
Jul 17 14:34:03.470682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.470659    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"]
Jul 17 14:34:03.637132 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.637102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13462214-8287-4730-aa49-e300f3a2c434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-7c94z\" (UID: \"13462214-8287-4730-aa49-e300f3a2c434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"
Jul 17 14:34:03.738029 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.738005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13462214-8287-4730-aa49-e300f3a2c434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-7c94z\" (UID: \"13462214-8287-4730-aa49-e300f3a2c434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"
Jul 17 14:34:03.738611 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.738592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13462214-8287-4730-aa49-e300f3a2c434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-7c94z\" (UID: \"13462214-8287-4730-aa49-e300f3a2c434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"
Jul 17 14:34:03.768798 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.768777    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"
Jul 17 14:34:03.897174 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:03.897146    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"]
Jul 17 14:34:03.899854 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:34:03.899826    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13462214_8287_4730_aa49_e300f3a2c434.slice/crio-edb856ac2f94fc8b91e780dd2e53d3db8f2baa94cbdbc5fd3f70d9bee99196ac WatchSource:0}: Error finding container edb856ac2f94fc8b91e780dd2e53d3db8f2baa94cbdbc5fd3f70d9bee99196ac: Status 404 returned error can't find the container with id edb856ac2f94fc8b91e780dd2e53d3db8f2baa94cbdbc5fd3f70d9bee99196ac
Jul 17 14:34:04.841735 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:04.841662    2575 generic.go:356] "Generic (PLEG): container finished" podID="13462214-8287-4730-aa49-e300f3a2c434" containerID="7bdc515d6341bb085f031cd2f902a782b4aae142014afd3a0bdce86204be80f4" exitCode=6
Jul 17 14:34:04.842037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:04.841738    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" event={"ID":"13462214-8287-4730-aa49-e300f3a2c434","Type":"ContainerDied","Data":"7bdc515d6341bb085f031cd2f902a782b4aae142014afd3a0bdce86204be80f4"}
Jul 17 14:34:04.842037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:04.841787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" event={"ID":"13462214-8287-4730-aa49-e300f3a2c434","Type":"ContainerStarted","Data":"edb856ac2f94fc8b91e780dd2e53d3db8f2baa94cbdbc5fd3f70d9bee99196ac"}
Jul 17 14:34:04.842037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:04.841895    2575 scope.go:117] "RemoveContainer" containerID="7bdc515d6341bb085f031cd2f902a782b4aae142014afd3a0bdce86204be80f4"
Jul 17 14:34:05.848753 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:05.848718    2575 generic.go:356] "Generic (PLEG): container finished" podID="13462214-8287-4730-aa49-e300f3a2c434" containerID="8f8e838bcf1e51f274356a706837d86a09650ad8a4b79e3fc00f19348ebf1547" exitCode=6
Jul 17 14:34:05.849170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:05.848811    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" event={"ID":"13462214-8287-4730-aa49-e300f3a2c434","Type":"ContainerDied","Data":"8f8e838bcf1e51f274356a706837d86a09650ad8a4b79e3fc00f19348ebf1547"}
Jul 17 14:34:05.849170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:05.848872    2575 scope.go:117] "RemoveContainer" containerID="7bdc515d6341bb085f031cd2f902a782b4aae142014afd3a0bdce86204be80f4"
Jul 17 14:34:05.849170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:05.849098    2575 scope.go:117] "RemoveContainer" containerID="8f8e838bcf1e51f274356a706837d86a09650ad8a4b79e3fc00f19348ebf1547"
Jul 17 14:34:05.849369 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:05.849351    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-7c94z_odh-ai-gateway-infra(13462214-8287-4730-aa49-e300f3a2c434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" podUID="13462214-8287-4730-aa49-e300f3a2c434"
Jul 17 14:34:06.855109 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:06.855082    2575 scope.go:117] "RemoveContainer" containerID="8f8e838bcf1e51f274356a706837d86a09650ad8a4b79e3fc00f19348ebf1547"
Jul 17 14:34:06.855482 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:06.855310    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-7c94z_odh-ai-gateway-infra(13462214-8287-4730-aa49-e300f3a2c434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" podUID="13462214-8287-4730-aa49-e300f3a2c434"
Jul 17 14:34:09.069504 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.069473    2575 scope.go:117] "RemoveContainer" containerID="455eb2079ee1b24412ea29858ab3e45337e71936c92e581a48bcf03d8dd39423"
Jul 17 14:34:09.432699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.432626    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"]
Jul 17 14:34:09.436090 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.436074    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.438891 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.438870    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-be4c79-serving-cert\""
Jul 17 14:34:09.448360 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.448334    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"]
Jul 17 14:34:09.588105 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.588081    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f67a1dc-6ba4-4580-bde2-391926c7efa9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.588105 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.588111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27wk\" (UniqueName: \"kubernetes.io/projected/0f67a1dc-6ba4-4580-bde2-391926c7efa9-kube-api-access-h27wk\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.588324 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.588175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.688644 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.688575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.688781 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.688689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f67a1dc-6ba4-4580-bde2-391926c7efa9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.688781 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:09.688703    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-serving-cert: secret "maas-api-e2e-inf-a-be4c79-serving-cert" not found
Jul 17 14:34:09.688781 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.688711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27wk\" (UniqueName: \"kubernetes.io/projected/0f67a1dc-6ba4-4580-bde2-391926c7efa9-kube-api-access-h27wk\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.688933 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:09.688777    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls podName:0f67a1dc-6ba4-4580-bde2-391926c7efa9 nodeName:}" failed. No retries permitted until 2026-07-17 14:34:10.188754535 +0000 UTC m=+2928.576101970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls") pod "maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk" (UID: "0f67a1dc-6ba4-4580-bde2-391926c7efa9") : secret "maas-api-e2e-inf-a-be4c79-serving-cert" not found
Jul 17 14:34:09.689320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.689303    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f67a1dc-6ba4-4580-bde2-391926c7efa9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.699478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.699461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27wk\" (UniqueName: \"kubernetes.io/projected/0f67a1dc-6ba4-4580-bde2-391926c7efa9-kube-api-access-h27wk\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:09.872519 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.872493    2575 generic.go:356] "Generic (PLEG): container finished" podID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerID="9d03dc107454448eeee7057e64c009ea5cf2fe68368869cbeb32c14b6360b581" exitCode=6
Jul 17 14:34:09.872669 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.872565    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" event={"ID":"2e0194f7-5cda-48a0-bd40-8631db137ecc","Type":"ContainerDied","Data":"9d03dc107454448eeee7057e64c009ea5cf2fe68368869cbeb32c14b6360b581"}
Jul 17 14:34:09.872669 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.872623    2575 scope.go:117] "RemoveContainer" containerID="455eb2079ee1b24412ea29858ab3e45337e71936c92e581a48bcf03d8dd39423"
Jul 17 14:34:09.872922 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:09.872906    2575 scope.go:117] "RemoveContainer" containerID="9d03dc107454448eeee7057e64c009ea5cf2fe68368869cbeb32c14b6360b581"
Jul 17 14:34:09.873148 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:09.873129    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z_odh-ai-gateway-infra(2e0194f7-5cda-48a0-bd40-8631db137ecc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc"
Jul 17 14:34:10.194259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:10.194226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:10.196505 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:10.196480    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls\") pod \"maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:10.346431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:10.346407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:10.499677 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:10.499654    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"]
Jul 17 14:34:10.501925 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:34:10.501896    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f67a1dc_6ba4_4580_bde2_391926c7efa9.slice/crio-a102b9ccddf2e60427b8b86402b5501588d6cd769b19438eac6eb2be60a062c8 WatchSource:0}: Error finding container a102b9ccddf2e60427b8b86402b5501588d6cd769b19438eac6eb2be60a062c8: Status 404 returned error can't find the container with id a102b9ccddf2e60427b8b86402b5501588d6cd769b19438eac6eb2be60a062c8
Jul 17 14:34:10.880618 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:10.880594    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk" event={"ID":"0f67a1dc-6ba4-4580-bde2-391926c7efa9","Type":"ContainerStarted","Data":"a102b9ccddf2e60427b8b86402b5501588d6cd769b19438eac6eb2be60a062c8"}
Jul 17 14:34:10.919619 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:10.919595    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"]
Jul 17 14:34:11.031674 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.031651    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"
Jul 17 14:34:11.101196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.101166    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0194f7-5cda-48a0-bd40-8631db137ecc-maas-api-service-ca\") pod \"2e0194f7-5cda-48a0-bd40-8631db137ecc\" (UID: \"2e0194f7-5cda-48a0-bd40-8631db137ecc\") "
Jul 17 14:34:11.101511 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.101489    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0194f7-5cda-48a0-bd40-8631db137ecc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e0194f7-5cda-48a0-bd40-8631db137ecc" (UID: "2e0194f7-5cda-48a0-bd40-8631db137ecc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:34:11.202711 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.202540    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0194f7-5cda-48a0-bd40-8631db137ecc-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:34:11.886031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.886002    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"
Jul 17 14:34:11.886169 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.886006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z" event={"ID":"2e0194f7-5cda-48a0-bd40-8631db137ecc","Type":"ContainerDied","Data":"1d7c5b0f4e43d92c7ef6654863efc8d8a32e53bdafd8c0cf4ff8e46e72eeaa8c"}
Jul 17 14:34:11.886169 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.886156    2575 scope.go:117] "RemoveContainer" containerID="9d03dc107454448eeee7057e64c009ea5cf2fe68368869cbeb32c14b6360b581"
Jul 17 14:34:11.887624 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.887599    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk" event={"ID":"0f67a1dc-6ba4-4580-bde2-391926c7efa9","Type":"ContainerStarted","Data":"9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1"}
Jul 17 14:34:11.887743 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.887652    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:11.909530 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.909490    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk" podStartSLOduration=2.542200787 podStartE2EDuration="2.909476751s" totalImagesPullingTime="367.275964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:34:11.907739049 +0000 UTC m=+2930.295086489" watchObservedRunningTime="2026-07-17 14:34:11.909476751 +0000 UTC m=+2930.296824190"
Jul 17 14:34:11.926600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.926575    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"]
Jul 17 14:34:11.934320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:11.934299    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-6wh9z"]
Jul 17 14:34:12.080917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:12.080889    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" path="/var/lib/kubelet/pods/2e0194f7-5cda-48a0-bd40-8631db137ecc/volumes"
Jul 17 14:34:17.542839 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.542804    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"]
Jul 17 14:34:17.543378 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543361    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.543439 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543382    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.543439 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543401    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.543439 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543411    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.543439 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543427    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.543439 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543435    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.543673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543514    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.543673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.543524    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:17.548318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.548290    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.551211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.551188    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b9f570-serving-cert\""
Jul 17 14:34:17.559817 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.559795    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"]
Jul 17 14:34:17.651871 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.651837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05054f99-c556-46ad-a7a0-06e0461502f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.652054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.651881    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskkt\" (UniqueName: \"kubernetes.io/projected/05054f99-c556-46ad-a7a0-06e0461502f2-kube-api-access-nskkt\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.652054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.651949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.753228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.753193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.753399 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.753319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05054f99-c556-46ad-a7a0-06e0461502f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.753399 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.753353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskkt\" (UniqueName: \"kubernetes.io/projected/05054f99-c556-46ad-a7a0-06e0461502f2-kube-api-access-nskkt\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.753399 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:17.753366    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-serving-cert: secret "maas-api-e2e-inf-b-b9f570-serving-cert" not found
Jul 17 14:34:17.753518 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:17.753441    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls podName:05054f99-c556-46ad-a7a0-06e0461502f2 nodeName:}" failed. No retries permitted until 2026-07-17 14:34:18.253419887 +0000 UTC m=+2936.640767319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls") pod "maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45" (UID: "05054f99-c556-46ad-a7a0-06e0461502f2") : secret "maas-api-e2e-inf-b-b9f570-serving-cert" not found
Jul 17 14:34:17.753924 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.753904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05054f99-c556-46ad-a7a0-06e0461502f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.774022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.773990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskkt\" (UniqueName: \"kubernetes.io/projected/05054f99-c556-46ad-a7a0-06e0461502f2-kube-api-access-nskkt\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:17.898460 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:17.898405    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:34:18.256980 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:18.256940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:18.259125 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:18.259105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:18.459732 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:18.459702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:18.600030 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:18.600005    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"]
Jul 17 14:34:18.601812 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:34:18.601782    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05054f99_c556_46ad_a7a0_06e0461502f2.slice/crio-1e03e02a240d641ec575115acbf8180018ce8cb50b576b6c37ea6f5a2d3989b4 WatchSource:0}: Error finding container 1e03e02a240d641ec575115acbf8180018ce8cb50b576b6c37ea6f5a2d3989b4: Status 404 returned error can't find the container with id 1e03e02a240d641ec575115acbf8180018ce8cb50b576b6c37ea6f5a2d3989b4
Jul 17 14:34:18.921860 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:18.921790    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45" event={"ID":"05054f99-c556-46ad-a7a0-06e0461502f2","Type":"ContainerStarted","Data":"1e03e02a240d641ec575115acbf8180018ce8cb50b576b6c37ea6f5a2d3989b4"}
Jul 17 14:34:19.292520 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.292491    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"]
Jul 17 14:34:19.293006 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.292992    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0194f7-5cda-48a0-bd40-8631db137ecc" containerName="revoke-keys"
Jul 17 14:34:19.296241 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.296226    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"
Jul 17 14:34:19.305516 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.305494    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"]
Jul 17 14:34:19.367687 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.367653    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a201a214-17e8-4c89-9250-e621bd4e0474-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb\" (UID: \"a201a214-17e8-4c89-9250-e621bd4e0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"
Jul 17 14:34:19.468744 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.468719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a201a214-17e8-4c89-9250-e621bd4e0474-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb\" (UID: \"a201a214-17e8-4c89-9250-e621bd4e0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"
Jul 17 14:34:19.469308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.469271    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a201a214-17e8-4c89-9250-e621bd4e0474-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb\" (UID: \"a201a214-17e8-4c89-9250-e621bd4e0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"
Jul 17 14:34:19.606922 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.606852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"
Jul 17 14:34:19.743492 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.743467    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"]
Jul 17 14:34:19.745809 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:34:19.745773    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda201a214_17e8_4c89_9250_e621bd4e0474.slice/crio-63b843b2fc12b228eadf7613c0ddd7f2c52ab404be41387debbba2a7c41b3961 WatchSource:0}: Error finding container 63b843b2fc12b228eadf7613c0ddd7f2c52ab404be41387debbba2a7c41b3961: Status 404 returned error can't find the container with id 63b843b2fc12b228eadf7613c0ddd7f2c52ab404be41387debbba2a7c41b3961
Jul 17 14:34:19.928476 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.928445    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" event={"ID":"a201a214-17e8-4c89-9250-e621bd4e0474","Type":"ContainerStarted","Data":"63b843b2fc12b228eadf7613c0ddd7f2c52ab404be41387debbba2a7c41b3961"}
Jul 17 14:34:19.929861 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.929841    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45" event={"ID":"05054f99-c556-46ad-a7a0-06e0461502f2","Type":"ContainerStarted","Data":"cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74"}
Jul 17 14:34:19.929967 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.929950    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:19.956385 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:19.956339    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45" podStartSLOduration=2.6131529799999997 podStartE2EDuration="2.956326276s" totalImagesPullingTime="343.173296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:34:19.955065504 +0000 UTC m=+2938.342412945" watchObservedRunningTime="2026-07-17 14:34:19.956326276 +0000 UTC m=+2938.343673716"
Jul 17 14:34:20.068986 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.068967    2575 scope.go:117] "RemoveContainer" containerID="8f8e838bcf1e51f274356a706837d86a09650ad8a4b79e3fc00f19348ebf1547"
Jul 17 14:34:20.937264 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.937187    2575 generic.go:356] "Generic (PLEG): container finished" podID="13462214-8287-4730-aa49-e300f3a2c434" containerID="f59f810f3b591a1fec6461e288279800555ac76841f3bc8e5bf6dd9f27e7f44d" exitCode=6
Jul 17 14:34:20.937616 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.937263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" event={"ID":"13462214-8287-4730-aa49-e300f3a2c434","Type":"ContainerDied","Data":"f59f810f3b591a1fec6461e288279800555ac76841f3bc8e5bf6dd9f27e7f44d"}
Jul 17 14:34:20.937616 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.937332    2575 scope.go:117] "RemoveContainer" containerID="8f8e838bcf1e51f274356a706837d86a09650ad8a4b79e3fc00f19348ebf1547"
Jul 17 14:34:20.937616 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.937603    2575 scope.go:117] "RemoveContainer" containerID="f59f810f3b591a1fec6461e288279800555ac76841f3bc8e5bf6dd9f27e7f44d"
Jul 17 14:34:20.937817 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:20.937796    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-7c94z_odh-ai-gateway-infra(13462214-8287-4730-aa49-e300f3a2c434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" podUID="13462214-8287-4730-aa49-e300f3a2c434"
Jul 17 14:34:20.938835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.938816    2575 generic.go:356] "Generic (PLEG): container finished" podID="a201a214-17e8-4c89-9250-e621bd4e0474" containerID="13c1a8c1ef9a99217118f1a36bf22284e4d80b807d36c58bcbb53e11249516ab" exitCode=6
Jul 17 14:34:20.938896 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.938846    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" event={"ID":"a201a214-17e8-4c89-9250-e621bd4e0474","Type":"ContainerDied","Data":"13c1a8c1ef9a99217118f1a36bf22284e4d80b807d36c58bcbb53e11249516ab"}
Jul 17 14:34:20.939055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:20.939040    2575 scope.go:117] "RemoveContainer" containerID="13c1a8c1ef9a99217118f1a36bf22284e4d80b807d36c58bcbb53e11249516ab"
Jul 17 14:34:21.947376 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:21.947294    2575 generic.go:356] "Generic (PLEG): container finished" podID="a201a214-17e8-4c89-9250-e621bd4e0474" containerID="0ef36b6f01c4a3f4eaa4afaa7cc4b8727721ee37f67c306f0b6218b21d63d909" exitCode=6
Jul 17 14:34:21.947736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:21.947369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" event={"ID":"a201a214-17e8-4c89-9250-e621bd4e0474","Type":"ContainerDied","Data":"0ef36b6f01c4a3f4eaa4afaa7cc4b8727721ee37f67c306f0b6218b21d63d909"}
Jul 17 14:34:21.947736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:21.947420    2575 scope.go:117] "RemoveContainer" containerID="13c1a8c1ef9a99217118f1a36bf22284e4d80b807d36c58bcbb53e11249516ab"
Jul 17 14:34:21.947736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:21.947568    2575 scope.go:117] "RemoveContainer" containerID="0ef36b6f01c4a3f4eaa4afaa7cc4b8727721ee37f67c306f0b6218b21d63d909"
Jul 17 14:34:21.947874 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:21.947754    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb_odh-ai-gateway-infra(a201a214-17e8-4c89-9250-e621bd4e0474)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" podUID="a201a214-17e8-4c89-9250-e621bd4e0474"
Jul 17 14:34:21.965436 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:21.965418    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"]
Jul 17 14:34:22.102776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.102754    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"
Jul 17 14:34:22.189833 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.189803    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13462214-8287-4730-aa49-e300f3a2c434-maas-api-service-ca\") pod \"13462214-8287-4730-aa49-e300f3a2c434\" (UID: \"13462214-8287-4730-aa49-e300f3a2c434\") "
Jul 17 14:34:22.190116 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.190095    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13462214-8287-4730-aa49-e300f3a2c434-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13462214-8287-4730-aa49-e300f3a2c434" (UID: "13462214-8287-4730-aa49-e300f3a2c434"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:34:22.290723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.290695    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13462214-8287-4730-aa49-e300f3a2c434-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:34:22.952760 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.952734    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"
Jul 17 14:34:22.952760 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.952737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z" event={"ID":"13462214-8287-4730-aa49-e300f3a2c434","Type":"ContainerDied","Data":"edb856ac2f94fc8b91e780dd2e53d3db8f2baa94cbdbc5fd3f70d9bee99196ac"}
Jul 17 14:34:22.953351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.952807    2575 scope.go:117] "RemoveContainer" containerID="f59f810f3b591a1fec6461e288279800555ac76841f3bc8e5bf6dd9f27e7f44d"
Jul 17 14:34:22.954794 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.954776    2575 scope.go:117] "RemoveContainer" containerID="0ef36b6f01c4a3f4eaa4afaa7cc4b8727721ee37f67c306f0b6218b21d63d909"
Jul 17 14:34:22.955007 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:22.954985    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb_odh-ai-gateway-infra(a201a214-17e8-4c89-9250-e621bd4e0474)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" podUID="a201a214-17e8-4c89-9250-e621bd4e0474"
Jul 17 14:34:22.984519 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.984498    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"]
Jul 17 14:34:22.992166 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:22.992147    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-7c94z"]
Jul 17 14:34:24.074000 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:24.073965    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13462214-8287-4730-aa49-e300f3a2c434" path="/var/lib/kubelet/pods/13462214-8287-4730-aa49-e300f3a2c434/volumes"
Jul 17 14:34:25.943873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:25.943841    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:34:31.942686 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.942656    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"]
Jul 17 14:34:31.943065 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943052    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.943065 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943066    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.943142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943080    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.943142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943086    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.943142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943099    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.943142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943105    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.943263 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943166    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.943263 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.943174    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:31.947684 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.947662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"
Jul 17 14:34:31.956586 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:31.956560    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"]
Jul 17 14:34:32.069046 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:32.069018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3070d90b-ddd8-4cb1-984f-56da959a8dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-ljrkl\" (UID: \"3070d90b-ddd8-4cb1-984f-56da959a8dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"
Jul 17 14:34:32.169677 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:32.169646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3070d90b-ddd8-4cb1-984f-56da959a8dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-ljrkl\" (UID: \"3070d90b-ddd8-4cb1-984f-56da959a8dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"
Jul 17 14:34:32.170397 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:32.170371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3070d90b-ddd8-4cb1-984f-56da959a8dd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-ljrkl\" (UID: \"3070d90b-ddd8-4cb1-984f-56da959a8dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"
Jul 17 14:34:32.260099 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:32.260071    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"
Jul 17 14:34:32.386405 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:32.386379    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"]
Jul 17 14:34:32.388618 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:34:32.388592    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3070d90b_ddd8_4cb1_984f_56da959a8dd7.slice/crio-9df81e82a10325caa9c0c0cceaf0f017af5380dbf7c5f92e5814248a2a538663 WatchSource:0}: Error finding container 9df81e82a10325caa9c0c0cceaf0f017af5380dbf7c5f92e5814248a2a538663: Status 404 returned error can't find the container with id 9df81e82a10325caa9c0c0cceaf0f017af5380dbf7c5f92e5814248a2a538663
Jul 17 14:34:33.006415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:33.006384    2575 generic.go:356] "Generic (PLEG): container finished" podID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerID="1ec653e588abfec2780f8d66d8c3288326dc712feacf87fcebc822b31c25e582" exitCode=6
Jul 17 14:34:33.006721 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:33.006462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" event={"ID":"3070d90b-ddd8-4cb1-984f-56da959a8dd7","Type":"ContainerDied","Data":"1ec653e588abfec2780f8d66d8c3288326dc712feacf87fcebc822b31c25e582"}
Jul 17 14:34:33.006721 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:33.006510    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" event={"ID":"3070d90b-ddd8-4cb1-984f-56da959a8dd7","Type":"ContainerStarted","Data":"9df81e82a10325caa9c0c0cceaf0f017af5380dbf7c5f92e5814248a2a538663"}
Jul 17 14:34:33.006721 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:33.006643    2575 scope.go:117] "RemoveContainer" containerID="1ec653e588abfec2780f8d66d8c3288326dc712feacf87fcebc822b31c25e582"
Jul 17 14:34:34.013566 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:34.013535    2575 generic.go:356] "Generic (PLEG): container finished" podID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerID="26aeda939c28ed8a92e3fc6fb865f1ce2a96917803f6c5d384f5ca9199f59c35" exitCode=6
Jul 17 14:34:34.013967 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:34.013625    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" event={"ID":"3070d90b-ddd8-4cb1-984f-56da959a8dd7","Type":"ContainerDied","Data":"26aeda939c28ed8a92e3fc6fb865f1ce2a96917803f6c5d384f5ca9199f59c35"}
Jul 17 14:34:34.013967 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:34.013712    2575 scope.go:117] "RemoveContainer" containerID="1ec653e588abfec2780f8d66d8c3288326dc712feacf87fcebc822b31c25e582"
Jul 17 14:34:34.013967 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:34.013784    2575 scope.go:117] "RemoveContainer" containerID="26aeda939c28ed8a92e3fc6fb865f1ce2a96917803f6c5d384f5ca9199f59c35"
Jul 17 14:34:34.014097 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:34.013999    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-ljrkl_odh-ai-gateway-infra(3070d90b-ddd8-4cb1-984f-56da959a8dd7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7"
Jul 17 14:34:35.019996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:35.019969    2575 scope.go:117] "RemoveContainer" containerID="26aeda939c28ed8a92e3fc6fb865f1ce2a96917803f6c5d384f5ca9199f59c35"
Jul 17 14:34:35.020455 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:35.020142    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-ljrkl_odh-ai-gateway-infra(3070d90b-ddd8-4cb1-984f-56da959a8dd7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7"
Jul 17 14:34:36.069346 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:36.069308    2575 scope.go:117] "RemoveContainer" containerID="0ef36b6f01c4a3f4eaa4afaa7cc4b8727721ee37f67c306f0b6218b21d63d909"
Jul 17 14:34:37.030530 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:37.030499    2575 generic.go:356] "Generic (PLEG): container finished" podID="a201a214-17e8-4c89-9250-e621bd4e0474" containerID="2c89624bd6a2b78b0b212bd1b75231e40c7f59b0a3824ee7911937a8d324e5e8" exitCode=6
Jul 17 14:34:37.030685 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:37.030533    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" event={"ID":"a201a214-17e8-4c89-9250-e621bd4e0474","Type":"ContainerDied","Data":"2c89624bd6a2b78b0b212bd1b75231e40c7f59b0a3824ee7911937a8d324e5e8"}
Jul 17 14:34:37.030685 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:37.030586    2575 scope.go:117] "RemoveContainer" containerID="0ef36b6f01c4a3f4eaa4afaa7cc4b8727721ee37f67c306f0b6218b21d63d909"
Jul 17 14:34:37.030881 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:37.030861    2575 scope.go:117] "RemoveContainer" containerID="2c89624bd6a2b78b0b212bd1b75231e40c7f59b0a3824ee7911937a8d324e5e8"
Jul 17 14:34:37.031091 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:37.031066    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb_odh-ai-gateway-infra(a201a214-17e8-4c89-9250-e621bd4e0474)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" podUID="a201a214-17e8-4c89-9250-e621bd4e0474"
Jul 17 14:34:38.060715 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:38.060684    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"]
Jul 17 14:34:38.191459 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:38.191437    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"
Jul 17 14:34:38.318095 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:38.318030    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a201a214-17e8-4c89-9250-e621bd4e0474-maas-api-service-ca\") pod \"a201a214-17e8-4c89-9250-e621bd4e0474\" (UID: \"a201a214-17e8-4c89-9250-e621bd4e0474\") "
Jul 17 14:34:38.318371 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:38.318352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a201a214-17e8-4c89-9250-e621bd4e0474-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a201a214-17e8-4c89-9250-e621bd4e0474" (UID: "a201a214-17e8-4c89-9250-e621bd4e0474"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:34:38.419173 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:38.419143    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a201a214-17e8-4c89-9250-e621bd4e0474-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:34:39.043213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:39.043185    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb" event={"ID":"a201a214-17e8-4c89-9250-e621bd4e0474","Type":"ContainerDied","Data":"63b843b2fc12b228eadf7613c0ddd7f2c52ab404be41387debbba2a7c41b3961"}
Jul 17 14:34:39.043213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:39.043202    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"
Jul 17 14:34:39.043458 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:39.043250    2575 scope.go:117] "RemoveContainer" containerID="2c89624bd6a2b78b0b212bd1b75231e40c7f59b0a3824ee7911937a8d324e5e8"
Jul 17 14:34:39.072154 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:39.072129    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"]
Jul 17 14:34:39.076207 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:39.076185    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-rhdmb"]
Jul 17 14:34:40.074829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:40.074795    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" path="/var/lib/kubelet/pods/a201a214-17e8-4c89-9250-e621bd4e0474/volumes"
Jul 17 14:34:42.659608 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.659580    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"]
Jul 17 14:34:42.659991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.659945    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:42.659991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.659956    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:42.659991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.659976    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:42.659991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.659981    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:42.660119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.660070    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13462214-8287-4730-aa49-e300f3a2c434" containerName="revoke-keys"
Jul 17 14:34:42.660119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.660081    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:42.660119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.660087    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:42.660119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.660093    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:42.664913 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.664898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"
Jul 17 14:34:42.673995 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.673959    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"]
Jul 17 14:34:42.753731 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.753706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130a52-c813-4964-89b3-1ae887bb9099-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9\" (UID: \"d8130a52-c813-4964-89b3-1ae887bb9099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"
Jul 17 14:34:42.854248 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.854220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130a52-c813-4964-89b3-1ae887bb9099-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9\" (UID: \"d8130a52-c813-4964-89b3-1ae887bb9099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"
Jul 17 14:34:42.854796 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.854777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130a52-c813-4964-89b3-1ae887bb9099-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9\" (UID: \"d8130a52-c813-4964-89b3-1ae887bb9099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"
Jul 17 14:34:42.976022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:42.975990    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"
Jul 17 14:34:43.109693 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:43.109663    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"]
Jul 17 14:34:43.112269 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:34:43.112240    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8130a52_c813_4964_89b3_1ae887bb9099.slice/crio-9b4ac8e03ec67d398a1da6451a69835680904386de72eab1f823a76049c09e44 WatchSource:0}: Error finding container 9b4ac8e03ec67d398a1da6451a69835680904386de72eab1f823a76049c09e44: Status 404 returned error can't find the container with id 9b4ac8e03ec67d398a1da6451a69835680904386de72eab1f823a76049c09e44
Jul 17 14:34:44.069868 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:44.069838    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8130a52-c813-4964-89b3-1ae887bb9099" containerID="4a5ca384fd0d63ff24388434f18de8af85ac9d1787842b3d44c39c8ca9d64141" exitCode=6
Jul 17 14:34:44.070218 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:44.070115    2575 scope.go:117] "RemoveContainer" containerID="4a5ca384fd0d63ff24388434f18de8af85ac9d1787842b3d44c39c8ca9d64141"
Jul 17 14:34:44.074548 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:44.074525    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" event={"ID":"d8130a52-c813-4964-89b3-1ae887bb9099","Type":"ContainerDied","Data":"4a5ca384fd0d63ff24388434f18de8af85ac9d1787842b3d44c39c8ca9d64141"}
Jul 17 14:34:44.074663 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:44.074563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" event={"ID":"d8130a52-c813-4964-89b3-1ae887bb9099","Type":"ContainerStarted","Data":"9b4ac8e03ec67d398a1da6451a69835680904386de72eab1f823a76049c09e44"}
Jul 17 14:34:45.076597 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:45.076564    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8130a52-c813-4964-89b3-1ae887bb9099" containerID="2f9102c6a57ef93511640ebc1eb7b22947c15648c7a0ac48fc6417d99dc9bbc6" exitCode=6
Jul 17 14:34:45.077061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:45.076608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" event={"ID":"d8130a52-c813-4964-89b3-1ae887bb9099","Type":"ContainerDied","Data":"2f9102c6a57ef93511640ebc1eb7b22947c15648c7a0ac48fc6417d99dc9bbc6"}
Jul 17 14:34:45.077061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:45.076689    2575 scope.go:117] "RemoveContainer" containerID="4a5ca384fd0d63ff24388434f18de8af85ac9d1787842b3d44c39c8ca9d64141"
Jul 17 14:34:45.077061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:45.076870    2575 scope.go:117] "RemoveContainer" containerID="2f9102c6a57ef93511640ebc1eb7b22947c15648c7a0ac48fc6417d99dc9bbc6"
Jul 17 14:34:45.077226 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:45.077098    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9_odh-ai-gateway-infra(d8130a52-c813-4964-89b3-1ae887bb9099)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" podUID="d8130a52-c813-4964-89b3-1ae887bb9099"
Jul 17 14:34:46.082810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:46.082787    2575 scope.go:117] "RemoveContainer" containerID="2f9102c6a57ef93511640ebc1eb7b22947c15648c7a0ac48fc6417d99dc9bbc6"
Jul 17 14:34:46.083188 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:46.082949    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9_odh-ai-gateway-infra(d8130a52-c813-4964-89b3-1ae887bb9099)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" podUID="d8130a52-c813-4964-89b3-1ae887bb9099"
Jul 17 14:34:49.069082 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:49.069058    2575 scope.go:117] "RemoveContainer" containerID="26aeda939c28ed8a92e3fc6fb865f1ce2a96917803f6c5d384f5ca9199f59c35"
Jul 17 14:34:50.103750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:50.103717    2575 generic.go:356] "Generic (PLEG): container finished" podID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerID="3984f2262014e03cad54618313c826206a7def258887f8720116007d8842e552" exitCode=6
Jul 17 14:34:50.104187 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:50.103775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" event={"ID":"3070d90b-ddd8-4cb1-984f-56da959a8dd7","Type":"ContainerDied","Data":"3984f2262014e03cad54618313c826206a7def258887f8720116007d8842e552"}
Jul 17 14:34:50.104187 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:50.103828    2575 scope.go:117] "RemoveContainer" containerID="26aeda939c28ed8a92e3fc6fb865f1ce2a96917803f6c5d384f5ca9199f59c35"
Jul 17 14:34:50.104187 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:50.104137    2575 scope.go:117] "RemoveContainer" containerID="3984f2262014e03cad54618313c826206a7def258887f8720116007d8842e552"
Jul 17 14:34:50.104366 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:50.104344    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-ljrkl_odh-ai-gateway-infra(3070d90b-ddd8-4cb1-984f-56da959a8dd7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7"
Jul 17 14:34:51.132771 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:51.132742    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"]
Jul 17 14:34:51.263603 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:51.263579    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"
Jul 17 14:34:51.328226 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:51.328196    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3070d90b-ddd8-4cb1-984f-56da959a8dd7-maas-api-service-ca\") pod \"3070d90b-ddd8-4cb1-984f-56da959a8dd7\" (UID: \"3070d90b-ddd8-4cb1-984f-56da959a8dd7\") "
Jul 17 14:34:51.328527 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:51.328507    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3070d90b-ddd8-4cb1-984f-56da959a8dd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3070d90b-ddd8-4cb1-984f-56da959a8dd7" (UID: "3070d90b-ddd8-4cb1-984f-56da959a8dd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:34:51.429078 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:51.429024    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3070d90b-ddd8-4cb1-984f-56da959a8dd7-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:34:52.115996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:52.115975    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"
Jul 17 14:34:52.116135 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:52.116030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl" event={"ID":"3070d90b-ddd8-4cb1-984f-56da959a8dd7","Type":"ContainerDied","Data":"9df81e82a10325caa9c0c0cceaf0f017af5380dbf7c5f92e5814248a2a538663"}
Jul 17 14:34:52.116135 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:52.116074    2575 scope.go:117] "RemoveContainer" containerID="3984f2262014e03cad54618313c826206a7def258887f8720116007d8842e552"
Jul 17 14:34:52.134770 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:52.134747    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"]
Jul 17 14:34:52.138013 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:52.137995    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-ljrkl"]
Jul 17 14:34:53.511762 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.511731    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"]
Jul 17 14:34:53.512182 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512167    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512230 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512185    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512230 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512197    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512230 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512203    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512230 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512214    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:53.512230 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512220    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a201a214-17e8-4c89-9250-e621bd4e0474" containerName="revoke-keys"
Jul 17 14:34:53.512494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512234    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512239    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512322    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512331    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.512494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.512345    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" containerName="revoke-keys"
Jul 17 14:34:53.515409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.515391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"
Jul 17 14:34:53.523322 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.523299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"]
Jul 17 14:34:53.647434 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.647409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29562cb-d8fa-4500-95f9-b61047dd4a60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-tjkst\" (UID: \"e29562cb-d8fa-4500-95f9-b61047dd4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"
Jul 17 14:34:53.747910 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.747885    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29562cb-d8fa-4500-95f9-b61047dd4a60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-tjkst\" (UID: \"e29562cb-d8fa-4500-95f9-b61047dd4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"
Jul 17 14:34:53.748477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.748459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29562cb-d8fa-4500-95f9-b61047dd4a60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-tjkst\" (UID: \"e29562cb-d8fa-4500-95f9-b61047dd4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"
Jul 17 14:34:53.825808 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.825760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"
Jul 17 14:34:53.955972 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:53.955949    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"]
Jul 17 14:34:53.957946 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:34:53.957921    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29562cb_d8fa_4500_95f9_b61047dd4a60.slice/crio-3bc173308fd395a04f51720a84e55115abcda0f1d9a2d02333baad5e06a8f2dd WatchSource:0}: Error finding container 3bc173308fd395a04f51720a84e55115abcda0f1d9a2d02333baad5e06a8f2dd: Status 404 returned error can't find the container with id 3bc173308fd395a04f51720a84e55115abcda0f1d9a2d02333baad5e06a8f2dd
Jul 17 14:34:54.075175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:54.075145    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3070d90b-ddd8-4cb1-984f-56da959a8dd7" path="/var/lib/kubelet/pods/3070d90b-ddd8-4cb1-984f-56da959a8dd7/volumes"
Jul 17 14:34:54.129572 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:54.129515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" event={"ID":"e29562cb-d8fa-4500-95f9-b61047dd4a60","Type":"ContainerStarted","Data":"3bc173308fd395a04f51720a84e55115abcda0f1d9a2d02333baad5e06a8f2dd"}
Jul 17 14:34:55.135747 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:55.135718    2575 generic.go:356] "Generic (PLEG): container finished" podID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerID="0a5180e084d5f62b98da6ec6635a2dc0ae882c8c80c1042d9334deb7cd863eb3" exitCode=6
Jul 17 14:34:55.136186 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:55.135794    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" event={"ID":"e29562cb-d8fa-4500-95f9-b61047dd4a60","Type":"ContainerDied","Data":"0a5180e084d5f62b98da6ec6635a2dc0ae882c8c80c1042d9334deb7cd863eb3"}
Jul 17 14:34:55.136186 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:55.135953    2575 scope.go:117] "RemoveContainer" containerID="0a5180e084d5f62b98da6ec6635a2dc0ae882c8c80c1042d9334deb7cd863eb3"
Jul 17 14:34:56.142267 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:56.142193    2575 generic.go:356] "Generic (PLEG): container finished" podID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerID="b1590709b64647d20c17f1a96adccc47e80158df64fe0ebf30c630067dbca005" exitCode=6
Jul 17 14:34:56.142642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:56.142298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" event={"ID":"e29562cb-d8fa-4500-95f9-b61047dd4a60","Type":"ContainerDied","Data":"b1590709b64647d20c17f1a96adccc47e80158df64fe0ebf30c630067dbca005"}
Jul 17 14:34:56.142642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:56.142353    2575 scope.go:117] "RemoveContainer" containerID="0a5180e084d5f62b98da6ec6635a2dc0ae882c8c80c1042d9334deb7cd863eb3"
Jul 17 14:34:56.142642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:56.142487    2575 scope.go:117] "RemoveContainer" containerID="b1590709b64647d20c17f1a96adccc47e80158df64fe0ebf30c630067dbca005"
Jul 17 14:34:56.142803 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:56.142685    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-tjkst_odh-ai-gateway-infra(e29562cb-d8fa-4500-95f9-b61047dd4a60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60"
Jul 17 14:34:57.148866 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:57.148838    2575 scope.go:117] "RemoveContainer" containerID="b1590709b64647d20c17f1a96adccc47e80158df64fe0ebf30c630067dbca005"
Jul 17 14:34:57.149224 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:34:57.149038    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-tjkst_odh-ai-gateway-infra(e29562cb-d8fa-4500-95f9-b61047dd4a60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60"
Jul 17 14:34:59.069179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:34:59.069151    2575 scope.go:117] "RemoveContainer" containerID="2f9102c6a57ef93511640ebc1eb7b22947c15648c7a0ac48fc6417d99dc9bbc6"
Jul 17 14:35:00.163610 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:00.163582    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8130a52-c813-4964-89b3-1ae887bb9099" containerID="47a86f43109b41c777c00c3c7acfec6a09aacadbc4f0baa0bee22e8760b91105" exitCode=6
Jul 17 14:35:00.164064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:00.163656    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" event={"ID":"d8130a52-c813-4964-89b3-1ae887bb9099","Type":"ContainerDied","Data":"47a86f43109b41c777c00c3c7acfec6a09aacadbc4f0baa0bee22e8760b91105"}
Jul 17 14:35:00.164064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:00.163730    2575 scope.go:117] "RemoveContainer" containerID="2f9102c6a57ef93511640ebc1eb7b22947c15648c7a0ac48fc6417d99dc9bbc6"
Jul 17 14:35:00.164064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:00.164045    2575 scope.go:117] "RemoveContainer" containerID="47a86f43109b41c777c00c3c7acfec6a09aacadbc4f0baa0bee22e8760b91105"
Jul 17 14:35:00.164263 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:00.164243    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9_odh-ai-gateway-infra(d8130a52-c813-4964-89b3-1ae887bb9099)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" podUID="d8130a52-c813-4964-89b3-1ae887bb9099"
Jul 17 14:35:01.198710 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:01.198678    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"]
Jul 17 14:35:01.316981 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:01.316960    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"
Jul 17 14:35:01.407204 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:01.407183    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130a52-c813-4964-89b3-1ae887bb9099-maas-api-service-ca\") pod \"d8130a52-c813-4964-89b3-1ae887bb9099\" (UID: \"d8130a52-c813-4964-89b3-1ae887bb9099\") "
Jul 17 14:35:01.407557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:01.407535    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8130a52-c813-4964-89b3-1ae887bb9099-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8130a52-c813-4964-89b3-1ae887bb9099" (UID: "d8130a52-c813-4964-89b3-1ae887bb9099"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:01.508523 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:01.508488    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130a52-c813-4964-89b3-1ae887bb9099-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:02.176052 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:02.176030    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"
Jul 17 14:35:02.176052 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:02.176045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9" event={"ID":"d8130a52-c813-4964-89b3-1ae887bb9099","Type":"ContainerDied","Data":"9b4ac8e03ec67d398a1da6451a69835680904386de72eab1f823a76049c09e44"}
Jul 17 14:35:02.176247 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:02.176097    2575 scope.go:117] "RemoveContainer" containerID="47a86f43109b41c777c00c3c7acfec6a09aacadbc4f0baa0bee22e8760b91105"
Jul 17 14:35:02.193271 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:02.193252    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"]
Jul 17 14:35:02.198897 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:02.198877    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-xwbq9"]
Jul 17 14:35:04.074673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:04.074644    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" path="/var/lib/kubelet/pods/d8130a52-c813-4964-89b3-1ae887bb9099/volumes"
Jul 17 14:35:09.069361 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.069333    2575 scope.go:117] "RemoveContainer" containerID="b1590709b64647d20c17f1a96adccc47e80158df64fe0ebf30c630067dbca005"
Jul 17 14:35:09.398672 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.398614    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"]
Jul 17 14:35:09.399011 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.398999    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.399052 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.399012    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.399052 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.399041    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.399052 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.399046    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.399170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.399054    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.399170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.399059    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.399170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.399127    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.399170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.399137    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:09.402401 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.402384    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"
Jul 17 14:35:09.414759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.414738    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"]
Jul 17 14:35:09.567877 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.567840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11779-b222-422f-a3d7-beea3c7bb07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb\" (UID: \"82c11779-b222-422f-a3d7-beea3c7bb07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"
Jul 17 14:35:09.669240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.669064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11779-b222-422f-a3d7-beea3c7bb07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb\" (UID: \"82c11779-b222-422f-a3d7-beea3c7bb07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"
Jul 17 14:35:09.669684 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.669661    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11779-b222-422f-a3d7-beea3c7bb07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb\" (UID: \"82c11779-b222-422f-a3d7-beea3c7bb07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"
Jul 17 14:35:09.712397 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.712377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"
Jul 17 14:35:09.851747 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:09.851724    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"]
Jul 17 14:35:09.853706 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:35:09.853680    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c11779_b222_422f_a3d7_beea3c7bb07c.slice/crio-85f039b095fe6f549563243f443356a4d00f277b554598706da0ca19fc366a9a WatchSource:0}: Error finding container 85f039b095fe6f549563243f443356a4d00f277b554598706da0ca19fc366a9a: Status 404 returned error can't find the container with id 85f039b095fe6f549563243f443356a4d00f277b554598706da0ca19fc366a9a
Jul 17 14:35:10.214099 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:10.214068    2575 generic.go:356] "Generic (PLEG): container finished" podID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerID="6009c6f66d735c9f71ee7c4f256fa2124fc883ff823f5d07ca003e4a7b0b81ae" exitCode=6
Jul 17 14:35:10.214409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:10.214139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" event={"ID":"e29562cb-d8fa-4500-95f9-b61047dd4a60","Type":"ContainerDied","Data":"6009c6f66d735c9f71ee7c4f256fa2124fc883ff823f5d07ca003e4a7b0b81ae"}
Jul 17 14:35:10.214409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:10.214210    2575 scope.go:117] "RemoveContainer" containerID="b1590709b64647d20c17f1a96adccc47e80158df64fe0ebf30c630067dbca005"
Jul 17 14:35:10.214521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:10.214499    2575 scope.go:117] "RemoveContainer" containerID="6009c6f66d735c9f71ee7c4f256fa2124fc883ff823f5d07ca003e4a7b0b81ae"
Jul 17 14:35:10.214711 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:10.214688    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-tjkst_odh-ai-gateway-infra(e29562cb-d8fa-4500-95f9-b61047dd4a60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60"
Jul 17 14:35:10.215680 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:10.215655    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" event={"ID":"82c11779-b222-422f-a3d7-beea3c7bb07c","Type":"ContainerStarted","Data":"f550f8a41edd4750525c33c4bd9192ffe26ef533b694459b7191fc40699998be"}
Jul 17 14:35:10.215772 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:10.215696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" event={"ID":"82c11779-b222-422f-a3d7-beea3c7bb07c","Type":"ContainerStarted","Data":"85f039b095fe6f549563243f443356a4d00f277b554598706da0ca19fc366a9a"}
Jul 17 14:35:10.259540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:10.259501    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" podStartSLOduration=1.259490115 podStartE2EDuration="1.259490115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:35:10.256673166 +0000 UTC m=+2988.644020605" watchObservedRunningTime="2026-07-17 14:35:10.259490115 +0000 UTC m=+2988.646837554"
Jul 17 14:35:11.222850 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.222818    2575 generic.go:356] "Generic (PLEG): container finished" podID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerID="f550f8a41edd4750525c33c4bd9192ffe26ef533b694459b7191fc40699998be" exitCode=6
Jul 17 14:35:11.222850 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.222852    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" event={"ID":"82c11779-b222-422f-a3d7-beea3c7bb07c","Type":"ContainerDied","Data":"f550f8a41edd4750525c33c4bd9192ffe26ef533b694459b7191fc40699998be"}
Jul 17 14:35:11.223245 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.223021    2575 scope.go:117] "RemoveContainer" containerID="f550f8a41edd4750525c33c4bd9192ffe26ef533b694459b7191fc40699998be"
Jul 17 14:35:11.255499 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.255466    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"]
Jul 17 14:35:11.417379 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.417357    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"
Jul 17 14:35:11.584224 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.584163    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29562cb-d8fa-4500-95f9-b61047dd4a60-maas-api-service-ca\") pod \"e29562cb-d8fa-4500-95f9-b61047dd4a60\" (UID: \"e29562cb-d8fa-4500-95f9-b61047dd4a60\") "
Jul 17 14:35:11.584492 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.584471    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29562cb-d8fa-4500-95f9-b61047dd4a60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e29562cb-d8fa-4500-95f9-b61047dd4a60" (UID: "e29562cb-d8fa-4500-95f9-b61047dd4a60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:11.685648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:11.685622    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29562cb-d8fa-4500-95f9-b61047dd4a60-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:12.229898 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.229874    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"
Jul 17 14:35:12.230266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.229874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst" event={"ID":"e29562cb-d8fa-4500-95f9-b61047dd4a60","Type":"ContainerDied","Data":"3bc173308fd395a04f51720a84e55115abcda0f1d9a2d02333baad5e06a8f2dd"}
Jul 17 14:35:12.230266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.230022    2575 scope.go:117] "RemoveContainer" containerID="6009c6f66d735c9f71ee7c4f256fa2124fc883ff823f5d07ca003e4a7b0b81ae"
Jul 17 14:35:12.232038 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.232013    2575 generic.go:356] "Generic (PLEG): container finished" podID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerID="07427469e66f98436cde6b8b448aa0f4ac2cb4c929f1cde47f7f3d13fe5b6c9b" exitCode=6
Jul 17 14:35:12.232195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.232144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" event={"ID":"82c11779-b222-422f-a3d7-beea3c7bb07c","Type":"ContainerDied","Data":"07427469e66f98436cde6b8b448aa0f4ac2cb4c929f1cde47f7f3d13fe5b6c9b"}
Jul 17 14:35:12.232304 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.232258    2575 scope.go:117] "RemoveContainer" containerID="07427469e66f98436cde6b8b448aa0f4ac2cb4c929f1cde47f7f3d13fe5b6c9b"
Jul 17 14:35:12.232446 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:12.232429    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb_odh-ai-gateway-infra(82c11779-b222-422f-a3d7-beea3c7bb07c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c"
Jul 17 14:35:12.238530 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.238508    2575 scope.go:117] "RemoveContainer" containerID="f550f8a41edd4750525c33c4bd9192ffe26ef533b694459b7191fc40699998be"
Jul 17 14:35:12.248936 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.248914    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"]
Jul 17 14:35:12.255838 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:12.255819    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-tjkst"]
Jul 17 14:35:13.239236 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:13.239208    2575 scope.go:117] "RemoveContainer" containerID="07427469e66f98436cde6b8b448aa0f4ac2cb4c929f1cde47f7f3d13fe5b6c9b"
Jul 17 14:35:13.239696 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:13.239442    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb_odh-ai-gateway-infra(82c11779-b222-422f-a3d7-beea3c7bb07c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c"
Jul 17 14:35:14.074974 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:14.074942    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" path="/var/lib/kubelet/pods/e29562cb-d8fa-4500-95f9-b61047dd4a60/volumes"
Jul 17 14:35:22.229237 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:22.229213    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:35:22.240397 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:22.240373    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:35:22.248440 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:22.248418    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:35:22.248584 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:22.248453    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:35:22.254486 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:22.254459    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:35:22.262810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:22.262791    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:35:26.069420 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:26.069393    2575 scope.go:117] "RemoveContainer" containerID="07427469e66f98436cde6b8b448aa0f4ac2cb4c929f1cde47f7f3d13fe5b6c9b"
Jul 17 14:35:27.309705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:27.309674    2575 generic.go:356] "Generic (PLEG): container finished" podID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerID="05e70774666661789d72bad3d77eb0c2afbdcbb5778bf250df86ce73b88db750" exitCode=6
Jul 17 14:35:27.310075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:27.309744    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" event={"ID":"82c11779-b222-422f-a3d7-beea3c7bb07c","Type":"ContainerDied","Data":"05e70774666661789d72bad3d77eb0c2afbdcbb5778bf250df86ce73b88db750"}
Jul 17 14:35:27.310075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:27.309800    2575 scope.go:117] "RemoveContainer" containerID="07427469e66f98436cde6b8b448aa0f4ac2cb4c929f1cde47f7f3d13fe5b6c9b"
Jul 17 14:35:27.310075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:27.310049    2575 scope.go:117] "RemoveContainer" containerID="05e70774666661789d72bad3d77eb0c2afbdcbb5778bf250df86ce73b88db750"
Jul 17 14:35:27.310243 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:27.310225    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb_odh-ai-gateway-infra(82c11779-b222-422f-a3d7-beea3c7bb07c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c"
Jul 17 14:35:28.338422 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:28.338388    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"]
Jul 17 14:35:28.468550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:28.468525    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"
Jul 17 14:35:28.505729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:28.505703    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11779-b222-422f-a3d7-beea3c7bb07c-maas-api-service-ca\") pod \"82c11779-b222-422f-a3d7-beea3c7bb07c\" (UID: \"82c11779-b222-422f-a3d7-beea3c7bb07c\") "
Jul 17 14:35:28.506021 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:28.505997    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c11779-b222-422f-a3d7-beea3c7bb07c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82c11779-b222-422f-a3d7-beea3c7bb07c" (UID: "82c11779-b222-422f-a3d7-beea3c7bb07c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:28.607064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:28.607007    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11779-b222-422f-a3d7-beea3c7bb07c-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:29.322097 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:29.322070    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"
Jul 17 14:35:29.322257 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:29.322067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb" event={"ID":"82c11779-b222-422f-a3d7-beea3c7bb07c","Type":"ContainerDied","Data":"85f039b095fe6f549563243f443356a4d00f277b554598706da0ca19fc366a9a"}
Jul 17 14:35:29.322257 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:29.322201    2575 scope.go:117] "RemoveContainer" containerID="05e70774666661789d72bad3d77eb0c2afbdcbb5778bf250df86ce73b88db750"
Jul 17 14:35:29.345239 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:29.345210    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"]
Jul 17 14:35:29.348368 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:29.348347    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-jmvsb"]
Jul 17 14:35:30.075027 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:30.074994    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" path="/var/lib/kubelet/pods/82c11779-b222-422f-a3d7-beea3c7bb07c/volumes"
Jul 17 14:35:31.995532 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.995496    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"]
Jul 17 14:35:31.996059 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996038    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:31.996152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996061    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:31.996152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996092    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996100    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996120    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:31.996152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996128    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:31.996152 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996147    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996156    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996165    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996173    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996259    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996270    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996298    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996310    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996318    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:31.996481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:31.996330    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8130a52-c813-4964-89b3-1ae887bb9099" containerName="revoke-keys"
Jul 17 14:35:32.001082 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.001062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"
Jul 17 14:35:32.004039 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.004023    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:35:32.010421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.010306    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"]
Jul 17 14:35:32.018653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.018631    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"]
Jul 17 14:35:32.018988 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.018974    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:32.018988 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.018987    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c11779-b222-422f-a3d7-beea3c7bb07c" containerName="revoke-keys"
Jul 17 14:35:32.019119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.019058    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29562cb-d8fa-4500-95f9-b61047dd4a60" containerName="revoke-keys"
Jul 17 14:35:32.021946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.021916    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"
Jul 17 14:35:32.032917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.032893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a15ace2-9e52-4706-91db-80a0a5a9745f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-sf8xh\" (UID: \"2a15ace2-9e52-4706-91db-80a0a5a9745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"
Jul 17 14:35:32.033040 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.033020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda460b-d8c8-47a7-a566-5003a448836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd\" (UID: \"ceda460b-d8c8-47a7-a566-5003a448836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"
Jul 17 14:35:32.034810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.034791    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"]
Jul 17 14:35:32.134345 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.134323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda460b-d8c8-47a7-a566-5003a448836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd\" (UID: \"ceda460b-d8c8-47a7-a566-5003a448836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"
Jul 17 14:35:32.134501 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.134375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a15ace2-9e52-4706-91db-80a0a5a9745f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-sf8xh\" (UID: \"2a15ace2-9e52-4706-91db-80a0a5a9745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"
Jul 17 14:35:32.134923 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.134903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda460b-d8c8-47a7-a566-5003a448836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd\" (UID: \"ceda460b-d8c8-47a7-a566-5003a448836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"
Jul 17 14:35:32.134963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.134911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a15ace2-9e52-4706-91db-80a0a5a9745f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-sf8xh\" (UID: \"2a15ace2-9e52-4706-91db-80a0a5a9745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"
Jul 17 14:35:32.313061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.312999    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"
Jul 17 14:35:32.331523 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.331501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"
Jul 17 14:35:32.458305 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.458254    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"]
Jul 17 14:35:32.462310 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:35:32.462123    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a15ace2_9e52_4706_91db_80a0a5a9745f.slice/crio-94c3e46f93743ec8f31bd8e1e5ac19f5712d8d8130e8f8dc09b60209fab1789e WatchSource:0}: Error finding container 94c3e46f93743ec8f31bd8e1e5ac19f5712d8d8130e8f8dc09b60209fab1789e: Status 404 returned error can't find the container with id 94c3e46f93743ec8f31bd8e1e5ac19f5712d8d8130e8f8dc09b60209fab1789e
Jul 17 14:35:32.486866 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.486842    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"]
Jul 17 14:35:32.494447 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:35:32.494424    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceda460b_d8c8_47a7_a566_5003a448836d.slice/crio-342b01f1c80ad78a9cb500ac351c4374b4a1d9aedc84bb5468fb8fafcfd17cda WatchSource:0}: Error finding container 342b01f1c80ad78a9cb500ac351c4374b4a1d9aedc84bb5468fb8fafcfd17cda: Status 404 returned error can't find the container with id 342b01f1c80ad78a9cb500ac351c4374b4a1d9aedc84bb5468fb8fafcfd17cda
Jul 17 14:35:32.721569 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.721541    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"]
Jul 17 14:35:32.724941 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.724910    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"
Jul 17 14:35:32.734031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.734005    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"]
Jul 17 14:35:32.738961 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.738930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325de1dc-1b0a-49c6-9fc7-0bc91fe49cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9\" (UID: \"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"
Jul 17 14:35:32.839590 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.839558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325de1dc-1b0a-49c6-9fc7-0bc91fe49cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9\" (UID: \"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"
Jul 17 14:35:32.840318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:32.840297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325de1dc-1b0a-49c6-9fc7-0bc91fe49cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9\" (UID: \"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"
Jul 17 14:35:33.038819 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.038794    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"
Jul 17 14:35:33.171994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.171878    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"]
Jul 17 14:35:33.200008 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:35:33.199984    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325de1dc_1b0a_49c6_9fc7_0bc91fe49cff.slice/crio-2c400d6bc67e61e0cb0aa1a10d657fa7e4db7a5e22af058160eae6dc44c538f7 WatchSource:0}: Error finding container 2c400d6bc67e61e0cb0aa1a10d657fa7e4db7a5e22af058160eae6dc44c538f7: Status 404 returned error can't find the container with id 2c400d6bc67e61e0cb0aa1a10d657fa7e4db7a5e22af058160eae6dc44c538f7
Jul 17 14:35:33.343569 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.343506    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" event={"ID":"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff","Type":"ContainerStarted","Data":"2c400d6bc67e61e0cb0aa1a10d657fa7e4db7a5e22af058160eae6dc44c538f7"}
Jul 17 14:35:33.344934 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.344911    2575 generic.go:356] "Generic (PLEG): container finished" podID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerID="4535aed4d4149dd271b5562c20e921a23ada43867e97cd65ae31eadbafef6c7a" exitCode=6
Jul 17 14:35:33.345045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.344999    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" event={"ID":"2a15ace2-9e52-4706-91db-80a0a5a9745f","Type":"ContainerDied","Data":"4535aed4d4149dd271b5562c20e921a23ada43867e97cd65ae31eadbafef6c7a"}
Jul 17 14:35:33.345106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.345050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" event={"ID":"2a15ace2-9e52-4706-91db-80a0a5a9745f","Type":"ContainerStarted","Data":"94c3e46f93743ec8f31bd8e1e5ac19f5712d8d8130e8f8dc09b60209fab1789e"}
Jul 17 14:35:33.345227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.345207    2575 scope.go:117] "RemoveContainer" containerID="4535aed4d4149dd271b5562c20e921a23ada43867e97cd65ae31eadbafef6c7a"
Jul 17 14:35:33.346623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.346600    2575 generic.go:356] "Generic (PLEG): container finished" podID="ceda460b-d8c8-47a7-a566-5003a448836d" containerID="6850fd31bc5e87f75046a7ccc42bd260b3db720d37641d552ef33a2b625031e5" exitCode=0
Jul 17 14:35:33.346708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.346643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd" event={"ID":"ceda460b-d8c8-47a7-a566-5003a448836d","Type":"ContainerDied","Data":"6850fd31bc5e87f75046a7ccc42bd260b3db720d37641d552ef33a2b625031e5"}
Jul 17 14:35:33.346708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:33.346680    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd" event={"ID":"ceda460b-d8c8-47a7-a566-5003a448836d","Type":"ContainerStarted","Data":"342b01f1c80ad78a9cb500ac351c4374b4a1d9aedc84bb5468fb8fafcfd17cda"}
Jul 17 14:35:34.353159 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.353125    2575 generic.go:356] "Generic (PLEG): container finished" podID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerID="da225dfbfd6d20928ae646d297045c14317a44368c94eb57f61c8cbbcdeb022d" exitCode=6
Jul 17 14:35:34.353582 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.353208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" event={"ID":"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff","Type":"ContainerDied","Data":"da225dfbfd6d20928ae646d297045c14317a44368c94eb57f61c8cbbcdeb022d"}
Jul 17 14:35:34.353582 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.353361    2575 scope.go:117] "RemoveContainer" containerID="da225dfbfd6d20928ae646d297045c14317a44368c94eb57f61c8cbbcdeb022d"
Jul 17 14:35:34.355091 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.355068    2575 generic.go:356] "Generic (PLEG): container finished" podID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerID="9be79221480ddce2be77022fe5413b6ea115bcb50f54cfa03cee74f3759ddc90" exitCode=6
Jul 17 14:35:34.355182 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.355107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" event={"ID":"2a15ace2-9e52-4706-91db-80a0a5a9745f","Type":"ContainerDied","Data":"9be79221480ddce2be77022fe5413b6ea115bcb50f54cfa03cee74f3759ddc90"}
Jul 17 14:35:34.355182 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.355157    2575 scope.go:117] "RemoveContainer" containerID="4535aed4d4149dd271b5562c20e921a23ada43867e97cd65ae31eadbafef6c7a"
Jul 17 14:35:34.355370 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.355355    2575 scope.go:117] "RemoveContainer" containerID="9be79221480ddce2be77022fe5413b6ea115bcb50f54cfa03cee74f3759ddc90"
Jul 17 14:35:34.355613 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:34.355542    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-sf8xh_odh-ai-gateway-infra(2a15ace2-9e52-4706-91db-80a0a5a9745f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f"
Jul 17 14:35:34.502328 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.502304    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"
Jul 17 14:35:34.554818 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.554791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda460b-d8c8-47a7-a566-5003a448836d-maas-api-service-ca\") pod \"ceda460b-d8c8-47a7-a566-5003a448836d\" (UID: \"ceda460b-d8c8-47a7-a566-5003a448836d\") "
Jul 17 14:35:34.555476 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.555425    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceda460b-d8c8-47a7-a566-5003a448836d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ceda460b-d8c8-47a7-a566-5003a448836d" (UID: "ceda460b-d8c8-47a7-a566-5003a448836d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:34.655579 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:34.655515    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda460b-d8c8-47a7-a566-5003a448836d-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:35.361324 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.361294    2575 generic.go:356] "Generic (PLEG): container finished" podID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerID="685e31c890010da99517ecbd2c273a7d89a77f16e206fe2e21858f66ad84c742" exitCode=6
Jul 17 14:35:35.361733 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.361362    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" event={"ID":"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff","Type":"ContainerDied","Data":"685e31c890010da99517ecbd2c273a7d89a77f16e206fe2e21858f66ad84c742"}
Jul 17 14:35:35.361733 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.361422    2575 scope.go:117] "RemoveContainer" containerID="da225dfbfd6d20928ae646d297045c14317a44368c94eb57f61c8cbbcdeb022d"
Jul 17 14:35:35.361733 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.361557    2575 scope.go:117] "RemoveContainer" containerID="685e31c890010da99517ecbd2c273a7d89a77f16e206fe2e21858f66ad84c742"
Jul 17 14:35:35.361896 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:35.361784    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9_odh-ai-gateway-infra(325de1dc-1b0a-49c6-9fc7-0bc91fe49cff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff"
Jul 17 14:35:35.363103 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.363086    2575 scope.go:117] "RemoveContainer" containerID="9be79221480ddce2be77022fe5413b6ea115bcb50f54cfa03cee74f3759ddc90"
Jul 17 14:35:35.363224 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:35.363208    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-sf8xh_odh-ai-gateway-infra(2a15ace2-9e52-4706-91db-80a0a5a9745f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f"
Jul 17 14:35:35.364016 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.363997    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd" event={"ID":"ceda460b-d8c8-47a7-a566-5003a448836d","Type":"ContainerDied","Data":"342b01f1c80ad78a9cb500ac351c4374b4a1d9aedc84bb5468fb8fafcfd17cda"}
Jul 17 14:35:35.364104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.364020    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"
Jul 17 14:35:35.364104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:35.364039    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342b01f1c80ad78a9cb500ac351c4374b4a1d9aedc84bb5468fb8fafcfd17cda"
Jul 17 14:35:36.370149 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:36.370121    2575 scope.go:117] "RemoveContainer" containerID="685e31c890010da99517ecbd2c273a7d89a77f16e206fe2e21858f66ad84c742"
Jul 17 14:35:36.370524 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:36.370289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9_odh-ai-gateway-infra(325de1dc-1b0a-49c6-9fc7-0bc91fe49cff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff"
Jul 17 14:35:42.524613 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.524580    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"]
Jul 17 14:35:42.525145 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.524825    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk" podUID="0f67a1dc-6ba4-4580-bde2-391926c7efa9" containerName="maas-api" containerID="cri-o://9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1" gracePeriod=30
Jul 17 14:35:42.769006 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.768983    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:35:42.819065 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.818996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f67a1dc-6ba4-4580-bde2-391926c7efa9-openshift-service-ca\") pod \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") "
Jul 17 14:35:42.819065 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.819049    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls\") pod \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") "
Jul 17 14:35:42.819238 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.819129    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h27wk\" (UniqueName: \"kubernetes.io/projected/0f67a1dc-6ba4-4580-bde2-391926c7efa9-kube-api-access-h27wk\") pod \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\" (UID: \"0f67a1dc-6ba4-4580-bde2-391926c7efa9\") "
Jul 17 14:35:42.819415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.819391    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f67a1dc-6ba4-4580-bde2-391926c7efa9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f67a1dc-6ba4-4580-bde2-391926c7efa9" (UID: "0f67a1dc-6ba4-4580-bde2-391926c7efa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:42.821576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.821551    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f67a1dc-6ba4-4580-bde2-391926c7efa9" (UID: "0f67a1dc-6ba4-4580-bde2-391926c7efa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:35:42.821681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.821553    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f67a1dc-6ba4-4580-bde2-391926c7efa9-kube-api-access-h27wk" (OuterVolumeSpecName: "kube-api-access-h27wk") pod "0f67a1dc-6ba4-4580-bde2-391926c7efa9" (UID: "0f67a1dc-6ba4-4580-bde2-391926c7efa9"). InnerVolumeSpecName "kube-api-access-h27wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:35:42.920521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.920500    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f67a1dc-6ba4-4580-bde2-391926c7efa9-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:42.920521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.920520    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h27wk\" (UniqueName: \"kubernetes.io/projected/0f67a1dc-6ba4-4580-bde2-391926c7efa9-kube-api-access-h27wk\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:42.920658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:42.920529    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f67a1dc-6ba4-4580-bde2-391926c7efa9-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:43.405620 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.405589    2575 generic.go:356] "Generic (PLEG): container finished" podID="0f67a1dc-6ba4-4580-bde2-391926c7efa9" containerID="9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1" exitCode=0
Jul 17 14:35:43.405777 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.405628    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"
Jul 17 14:35:43.405777 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.405652    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk" event={"ID":"0f67a1dc-6ba4-4580-bde2-391926c7efa9","Type":"ContainerDied","Data":"9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1"}
Jul 17 14:35:43.405777 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.405714    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk" event={"ID":"0f67a1dc-6ba4-4580-bde2-391926c7efa9","Type":"ContainerDied","Data":"a102b9ccddf2e60427b8b86402b5501588d6cd769b19438eac6eb2be60a062c8"}
Jul 17 14:35:43.407301 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.405744    2575 scope.go:117] "RemoveContainer" containerID="9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1"
Jul 17 14:35:43.422428 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.422412    2575 scope.go:117] "RemoveContainer" containerID="9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1"
Jul 17 14:35:43.422676 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:43.422656    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1\": container with ID starting with 9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1 not found: ID does not exist" containerID="9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1"
Jul 17 14:35:43.422723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.422685    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1"} err="failed to get container status \"9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1\": rpc error: code = NotFound desc = could not find container \"9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1\": container with ID starting with 9f03c68749c2e8c22927024e6cb134a913e4e1a83caaba01f61d85747f45d5f1 not found: ID does not exist"
Jul 17 14:35:43.431490 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.431469    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"]
Jul 17 14:35:43.435969 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.435949    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be4c79-57568d864d-sqcrk"]
Jul 17 14:35:43.572186 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.572159    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"]
Jul 17 14:35:43.572602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.572587    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceda460b-d8c8-47a7-a566-5003a448836d" containerName="revoke-keys"
Jul 17 14:35:43.572653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.572603    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceda460b-d8c8-47a7-a566-5003a448836d" containerName="revoke-keys"
Jul 17 14:35:43.572653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.572620    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f67a1dc-6ba4-4580-bde2-391926c7efa9" containerName="maas-api"
Jul 17 14:35:43.572653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.572628    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f67a1dc-6ba4-4580-bde2-391926c7efa9" containerName="maas-api"
Jul 17 14:35:43.572751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.572699    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceda460b-d8c8-47a7-a566-5003a448836d" containerName="revoke-keys"
Jul 17 14:35:43.572751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.572715    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f67a1dc-6ba4-4580-bde2-391926c7efa9" containerName="maas-api"
Jul 17 14:35:43.577378 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.577363    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"
Jul 17 14:35:43.584539 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.584517    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"]
Jul 17 14:35:43.626034 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.626010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a67ae3-b412-4bcc-97f8-bb2f718b5825-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf\" (UID: \"04a67ae3-b412-4bcc-97f8-bb2f718b5825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"
Jul 17 14:35:43.726693 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.726666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a67ae3-b412-4bcc-97f8-bb2f718b5825-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf\" (UID: \"04a67ae3-b412-4bcc-97f8-bb2f718b5825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"
Jul 17 14:35:43.727193 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.727177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a67ae3-b412-4bcc-97f8-bb2f718b5825-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf\" (UID: \"04a67ae3-b412-4bcc-97f8-bb2f718b5825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"
Jul 17 14:35:43.888272 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:43.888254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"
Jul 17 14:35:44.023023 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:44.023003    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"]
Jul 17 14:35:44.025372 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:35:44.025346    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a67ae3_b412_4bcc_97f8_bb2f718b5825.slice/crio-41a9d65223803c1285c7c1604c7a1113dbb2e60488cd1ef56c041c6413007b1f WatchSource:0}: Error finding container 41a9d65223803c1285c7c1604c7a1113dbb2e60488cd1ef56c041c6413007b1f: Status 404 returned error can't find the container with id 41a9d65223803c1285c7c1604c7a1113dbb2e60488cd1ef56c041c6413007b1f
Jul 17 14:35:44.074766 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:44.074742    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f67a1dc-6ba4-4580-bde2-391926c7efa9" path="/var/lib/kubelet/pods/0f67a1dc-6ba4-4580-bde2-391926c7efa9/volumes"
Jul 17 14:35:44.412398 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:44.412320    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" event={"ID":"04a67ae3-b412-4bcc-97f8-bb2f718b5825","Type":"ContainerStarted","Data":"84cc6c821317909ae1f82af01c4f61206800a8e2c1218eff1a64cc6d0107a1de"}
Jul 17 14:35:44.412398 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:44.412376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" event={"ID":"04a67ae3-b412-4bcc-97f8-bb2f718b5825","Type":"ContainerStarted","Data":"41a9d65223803c1285c7c1604c7a1113dbb2e60488cd1ef56c041c6413007b1f"}
Jul 17 14:35:44.429574 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:44.429529    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" podStartSLOduration=1.429514891 podStartE2EDuration="1.429514891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:35:44.428971333 +0000 UTC m=+3022.816318772" watchObservedRunningTime="2026-07-17 14:35:44.429514891 +0000 UTC m=+3022.816862332"
Jul 17 14:35:45.419182 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:45.419141    2575 generic.go:356] "Generic (PLEG): container finished" podID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerID="84cc6c821317909ae1f82af01c4f61206800a8e2c1218eff1a64cc6d0107a1de" exitCode=6
Jul 17 14:35:45.419602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:45.419200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" event={"ID":"04a67ae3-b412-4bcc-97f8-bb2f718b5825","Type":"ContainerDied","Data":"84cc6c821317909ae1f82af01c4f61206800a8e2c1218eff1a64cc6d0107a1de"}
Jul 17 14:35:45.419602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:45.419421    2575 scope.go:117] "RemoveContainer" containerID="84cc6c821317909ae1f82af01c4f61206800a8e2c1218eff1a64cc6d0107a1de"
Jul 17 14:35:46.069933 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:46.069905    2575 scope.go:117] "RemoveContainer" containerID="9be79221480ddce2be77022fe5413b6ea115bcb50f54cfa03cee74f3759ddc90"
Jul 17 14:35:46.426178 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:46.426114    2575 generic.go:356] "Generic (PLEG): container finished" podID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerID="3e2c8b4d845181e723e5fa05590d0a5a10ba6a24bf065e95d6c3a66fb22e51e8" exitCode=6
Jul 17 14:35:46.426178 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:46.426152    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" event={"ID":"04a67ae3-b412-4bcc-97f8-bb2f718b5825","Type":"ContainerDied","Data":"3e2c8b4d845181e723e5fa05590d0a5a10ba6a24bf065e95d6c3a66fb22e51e8"}
Jul 17 14:35:46.426580 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:46.426207    2575 scope.go:117] "RemoveContainer" containerID="84cc6c821317909ae1f82af01c4f61206800a8e2c1218eff1a64cc6d0107a1de"
Jul 17 14:35:46.426580 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:46.426407    2575 scope.go:117] "RemoveContainer" containerID="3e2c8b4d845181e723e5fa05590d0a5a10ba6a24bf065e95d6c3a66fb22e51e8"
Jul 17 14:35:46.426682 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:46.426606    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf_odh-ai-gateway-infra(04a67ae3-b412-4bcc-97f8-bb2f718b5825)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825"
Jul 17 14:35:46.428184 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:46.428167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" event={"ID":"2a15ace2-9e52-4706-91db-80a0a5a9745f","Type":"ContainerStarted","Data":"ab7b6d6d7bccc5f5e95ef38f0b314aef296bbb7a2864ce70b14d053345302414"}
Jul 17 14:35:46.459825 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:46.459779    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" podStartSLOduration=15.459767013 podStartE2EDuration="15.459767013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:35:46.457856413 +0000 UTC m=+3024.845203852" watchObservedRunningTime="2026-07-17 14:35:46.459767013 +0000 UTC m=+3024.847114453"
Jul 17 14:35:47.164138 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.164097    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"]
Jul 17 14:35:47.169324 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.169288    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be4c79-7wjxd"]
Jul 17 14:35:47.434987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.434909    2575 generic.go:356] "Generic (PLEG): container finished" podID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerID="ab7b6d6d7bccc5f5e95ef38f0b314aef296bbb7a2864ce70b14d053345302414" exitCode=6
Jul 17 14:35:47.434987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.434976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" event={"ID":"2a15ace2-9e52-4706-91db-80a0a5a9745f","Type":"ContainerDied","Data":"ab7b6d6d7bccc5f5e95ef38f0b314aef296bbb7a2864ce70b14d053345302414"}
Jul 17 14:35:47.435521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.435037    2575 scope.go:117] "RemoveContainer" containerID="9be79221480ddce2be77022fe5413b6ea115bcb50f54cfa03cee74f3759ddc90"
Jul 17 14:35:47.435521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.435397    2575 scope.go:117] "RemoveContainer" containerID="ab7b6d6d7bccc5f5e95ef38f0b314aef296bbb7a2864ce70b14d053345302414"
Jul 17 14:35:47.435676 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:47.435655    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-sf8xh_odh-ai-gateway-infra(2a15ace2-9e52-4706-91db-80a0a5a9745f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f"
Jul 17 14:35:47.436901 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.436885    2575 scope.go:117] "RemoveContainer" containerID="3e2c8b4d845181e723e5fa05590d0a5a10ba6a24bf065e95d6c3a66fb22e51e8"
Jul 17 14:35:47.437075 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:47.437053    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf_odh-ai-gateway-infra(04a67ae3-b412-4bcc-97f8-bb2f718b5825)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825"
Jul 17 14:35:47.474783 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:47.474758    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"]
Jul 17 14:35:48.075076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:48.075046    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceda460b-d8c8-47a7-a566-5003a448836d" path="/var/lib/kubelet/pods/ceda460b-d8c8-47a7-a566-5003a448836d/volumes"
Jul 17 14:35:48.569108 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:48.569086    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"
Jul 17 14:35:48.668298 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:48.668253    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a15ace2-9e52-4706-91db-80a0a5a9745f-maas-api-service-ca\") pod \"2a15ace2-9e52-4706-91db-80a0a5a9745f\" (UID: \"2a15ace2-9e52-4706-91db-80a0a5a9745f\") "
Jul 17 14:35:48.668552 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:48.668531    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a15ace2-9e52-4706-91db-80a0a5a9745f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a15ace2-9e52-4706-91db-80a0a5a9745f" (UID: "2a15ace2-9e52-4706-91db-80a0a5a9745f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:48.769735 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:48.769704    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a15ace2-9e52-4706-91db-80a0a5a9745f-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:49.069460 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.069394    2575 scope.go:117] "RemoveContainer" containerID="685e31c890010da99517ecbd2c273a7d89a77f16e206fe2e21858f66ad84c742"
Jul 17 14:35:49.449920 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.449853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" event={"ID":"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff","Type":"ContainerStarted","Data":"fdd5ce8002c912e09946d86811f0c670a5259be98927f137dba8c29691e9b7be"}
Jul 17 14:35:49.451115 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.451071    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"
Jul 17 14:35:49.451115 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.451080    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh" event={"ID":"2a15ace2-9e52-4706-91db-80a0a5a9745f","Type":"ContainerDied","Data":"94c3e46f93743ec8f31bd8e1e5ac19f5712d8d8130e8f8dc09b60209fab1789e"}
Jul 17 14:35:49.451242 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.451137    2575 scope.go:117] "RemoveContainer" containerID="ab7b6d6d7bccc5f5e95ef38f0b314aef296bbb7a2864ce70b14d053345302414"
Jul 17 14:35:49.470471 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.470425    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" podStartSLOduration=17.470413566 podStartE2EDuration="17.470413566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:35:49.467948901 +0000 UTC m=+3027.855296341" watchObservedRunningTime="2026-07-17 14:35:49.470413566 +0000 UTC m=+3027.857761006"
Jul 17 14:35:49.488334 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.488308    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"]
Jul 17 14:35:49.492796 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:49.492776    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sf8xh"]
Jul 17 14:35:50.074601 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:50.074572    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" path="/var/lib/kubelet/pods/2a15ace2-9e52-4706-91db-80a0a5a9745f/volumes"
Jul 17 14:35:50.458007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:50.457921    2575 generic.go:356] "Generic (PLEG): container finished" podID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerID="fdd5ce8002c912e09946d86811f0c670a5259be98927f137dba8c29691e9b7be" exitCode=6
Jul 17 14:35:50.458007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:50.457992    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" event={"ID":"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff","Type":"ContainerDied","Data":"fdd5ce8002c912e09946d86811f0c670a5259be98927f137dba8c29691e9b7be"}
Jul 17 14:35:50.458196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:50.458042    2575 scope.go:117] "RemoveContainer" containerID="685e31c890010da99517ecbd2c273a7d89a77f16e206fe2e21858f66ad84c742"
Jul 17 14:35:50.458316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:50.458300    2575 scope.go:117] "RemoveContainer" containerID="fdd5ce8002c912e09946d86811f0c670a5259be98927f137dba8c29691e9b7be"
Jul 17 14:35:50.458512 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:35:50.458484    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9_odh-ai-gateway-infra(325de1dc-1b0a-49c6-9fc7-0bc91fe49cff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff"
Jul 17 14:35:50.484795 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:50.484772    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"]
Jul 17 14:35:51.593151 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:51.593129    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"
Jul 17 14:35:51.693039 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:51.693010    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325de1dc-1b0a-49c6-9fc7-0bc91fe49cff-maas-api-service-ca\") pod \"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff\" (UID: \"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff\") "
Jul 17 14:35:51.693374 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:51.693354    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325de1dc-1b0a-49c6-9fc7-0bc91fe49cff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" (UID: "325de1dc-1b0a-49c6-9fc7-0bc91fe49cff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:51.794112 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:51.794048    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325de1dc-1b0a-49c6-9fc7-0bc91fe49cff-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:52.472326 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:52.472297    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"
Jul 17 14:35:52.472326 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:52.472308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9" event={"ID":"325de1dc-1b0a-49c6-9fc7-0bc91fe49cff","Type":"ContainerDied","Data":"2c400d6bc67e61e0cb0aa1a10d657fa7e4db7a5e22af058160eae6dc44c538f7"}
Jul 17 14:35:52.472557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:52.472369    2575 scope.go:117] "RemoveContainer" containerID="fdd5ce8002c912e09946d86811f0c670a5259be98927f137dba8c29691e9b7be"
Jul 17 14:35:52.508269 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:52.508246    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"]
Jul 17 14:35:52.512256 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:52.512237    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nm6s9"]
Jul 17 14:35:54.079144 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.079110    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" path="/var/lib/kubelet/pods/325de1dc-1b0a-49c6-9fc7-0bc91fe49cff/volumes"
Jul 17 14:35:54.566021 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.565993    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"]
Jul 17 14:35:54.566402 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566388    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.566450 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566403    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.566450 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566412    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:35:54.566450 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566418    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:35:54.566450 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566432    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.566450 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566437    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.566450 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566451    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:35:54.566635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566457    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:35:54.566635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566468    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.566635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566474    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.566635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566532    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.566635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566538    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:35:54.566635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566545    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:35:54.566635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.566554    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:35:54.570903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.570881    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"
Jul 17 14:35:54.584186 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.584162    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"]
Jul 17 14:35:54.615261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.615236    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74eca80a-45df-4e76-9ff4-64584697ffa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw\" (UID: \"74eca80a-45df-4e76-9ff4-64584697ffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"
Jul 17 14:35:54.716104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.716054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74eca80a-45df-4e76-9ff4-64584697ffa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw\" (UID: \"74eca80a-45df-4e76-9ff4-64584697ffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"
Jul 17 14:35:54.716697 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.716675    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74eca80a-45df-4e76-9ff4-64584697ffa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw\" (UID: \"74eca80a-45df-4e76-9ff4-64584697ffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"
Jul 17 14:35:54.881342 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:54.881257    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"
Jul 17 14:35:55.012705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:55.012676    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"]
Jul 17 14:35:55.015011 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:35:55.014986    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74eca80a_45df_4e76_9ff4_64584697ffa3.slice/crio-15450189d5646fa9bc5b65fc91d45ebbeb5dc85d108149ccd596038d95a98c8e WatchSource:0}: Error finding container 15450189d5646fa9bc5b65fc91d45ebbeb5dc85d108149ccd596038d95a98c8e: Status 404 returned error can't find the container with id 15450189d5646fa9bc5b65fc91d45ebbeb5dc85d108149ccd596038d95a98c8e
Jul 17 14:35:55.489951 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:55.489910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw" event={"ID":"74eca80a-45df-4e76-9ff4-64584697ffa3","Type":"ContainerStarted","Data":"b2e72c90b3dc2fb4a16b2225bc531ff247771786df053ef49407b1a19c9920f7"}
Jul 17 14:35:55.490354 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:55.489976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw" event={"ID":"74eca80a-45df-4e76-9ff4-64584697ffa3","Type":"ContainerStarted","Data":"15450189d5646fa9bc5b65fc91d45ebbeb5dc85d108149ccd596038d95a98c8e"}
Jul 17 14:35:55.506754 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:55.506703    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw" podStartSLOduration=1.506687466 podStartE2EDuration="1.506687466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:35:55.505789402 +0000 UTC m=+3033.893136841" watchObservedRunningTime="2026-07-17 14:35:55.506687466 +0000 UTC m=+3033.894034947"
Jul 17 14:35:56.495681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:56.495650    2575 generic.go:356] "Generic (PLEG): container finished" podID="74eca80a-45df-4e76-9ff4-64584697ffa3" containerID="b2e72c90b3dc2fb4a16b2225bc531ff247771786df053ef49407b1a19c9920f7" exitCode=0
Jul 17 14:35:56.496050 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:56.495710    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw" event={"ID":"74eca80a-45df-4e76-9ff4-64584697ffa3","Type":"ContainerDied","Data":"b2e72c90b3dc2fb4a16b2225bc531ff247771786df053ef49407b1a19c9920f7"}
Jul 17 14:35:57.630725 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:57.630704    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"
Jul 17 14:35:57.742711 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:57.742688    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74eca80a-45df-4e76-9ff4-64584697ffa3-maas-api-service-ca\") pod \"74eca80a-45df-4e76-9ff4-64584697ffa3\" (UID: \"74eca80a-45df-4e76-9ff4-64584697ffa3\") "
Jul 17 14:35:57.743037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:57.743012    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74eca80a-45df-4e76-9ff4-64584697ffa3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74eca80a-45df-4e76-9ff4-64584697ffa3" (UID: "74eca80a-45df-4e76-9ff4-64584697ffa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:35:57.843700 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:57.843642    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74eca80a-45df-4e76-9ff4-64584697ffa3-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:35:58.506399 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:58.506374    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"
Jul 17 14:35:58.506557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:58.506415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw" event={"ID":"74eca80a-45df-4e76-9ff4-64584697ffa3","Type":"ContainerDied","Data":"15450189d5646fa9bc5b65fc91d45ebbeb5dc85d108149ccd596038d95a98c8e"}
Jul 17 14:35:58.506557 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:58.506453    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15450189d5646fa9bc5b65fc91d45ebbeb5dc85d108149ccd596038d95a98c8e"
Jul 17 14:35:59.069021 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:59.068995    2575 scope.go:117] "RemoveContainer" containerID="3e2c8b4d845181e723e5fa05590d0a5a10ba6a24bf065e95d6c3a66fb22e51e8"
Jul 17 14:35:59.513224 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:35:59.513198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" event={"ID":"04a67ae3-b412-4bcc-97f8-bb2f718b5825","Type":"ContainerStarted","Data":"27c06a334f2d748526991bed6800d536dab80bfa6d2461ad4e1793215748d79e"}
Jul 17 14:36:00.520927 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:00.520888    2575 generic.go:356] "Generic (PLEG): container finished" podID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerID="27c06a334f2d748526991bed6800d536dab80bfa6d2461ad4e1793215748d79e" exitCode=6
Jul 17 14:36:00.521387 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:00.520935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" event={"ID":"04a67ae3-b412-4bcc-97f8-bb2f718b5825","Type":"ContainerDied","Data":"27c06a334f2d748526991bed6800d536dab80bfa6d2461ad4e1793215748d79e"}
Jul 17 14:36:00.521387 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:00.521024    2575 scope.go:117] "RemoveContainer" containerID="3e2c8b4d845181e723e5fa05590d0a5a10ba6a24bf065e95d6c3a66fb22e51e8"
Jul 17 14:36:00.521387 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:00.521342    2575 scope.go:117] "RemoveContainer" containerID="27c06a334f2d748526991bed6800d536dab80bfa6d2461ad4e1793215748d79e"
Jul 17 14:36:00.521552 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:00.521526    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf_odh-ai-gateway-infra(04a67ae3-b412-4bcc-97f8-bb2f718b5825)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825"
Jul 17 14:36:00.545010 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:00.544986    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"]
Jul 17 14:36:01.655327 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:01.655307    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"
Jul 17 14:36:01.775829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:01.775799    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a67ae3-b412-4bcc-97f8-bb2f718b5825-maas-api-service-ca\") pod \"04a67ae3-b412-4bcc-97f8-bb2f718b5825\" (UID: \"04a67ae3-b412-4bcc-97f8-bb2f718b5825\") "
Jul 17 14:36:01.776119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:01.776099    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a67ae3-b412-4bcc-97f8-bb2f718b5825-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04a67ae3-b412-4bcc-97f8-bb2f718b5825" (UID: "04a67ae3-b412-4bcc-97f8-bb2f718b5825"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:36:01.877258 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:01.877194    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a67ae3-b412-4bcc-97f8-bb2f718b5825-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:02.532748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:02.532719    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"
Jul 17 14:36:02.532903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:02.532757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf" event={"ID":"04a67ae3-b412-4bcc-97f8-bb2f718b5825","Type":"ContainerDied","Data":"41a9d65223803c1285c7c1604c7a1113dbb2e60488cd1ef56c041c6413007b1f"}
Jul 17 14:36:02.532903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:02.532817    2575 scope.go:117] "RemoveContainer" containerID="27c06a334f2d748526991bed6800d536dab80bfa6d2461ad4e1793215748d79e"
Jul 17 14:36:02.549267 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:02.549244    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"]
Jul 17 14:36:02.551717 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:02.551699    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-rzjvf"]
Jul 17 14:36:04.075056 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:04.075016    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" path="/var/lib/kubelet/pods/04a67ae3-b412-4bcc-97f8-bb2f718b5825/volumes"
Jul 17 14:36:05.060103 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.060060    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"]
Jul 17 14:36:05.060364 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.060340    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45" podUID="05054f99-c556-46ad-a7a0-06e0461502f2" containerName="maas-api" containerID="cri-o://cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74" gracePeriod=30
Jul 17 14:36:05.306912 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.306889    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:36:05.405887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.405820    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nskkt\" (UniqueName: \"kubernetes.io/projected/05054f99-c556-46ad-a7a0-06e0461502f2-kube-api-access-nskkt\") pod \"05054f99-c556-46ad-a7a0-06e0461502f2\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") "
Jul 17 14:36:05.405887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.405858    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls\") pod \"05054f99-c556-46ad-a7a0-06e0461502f2\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") "
Jul 17 14:36:05.406044 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.405907    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05054f99-c556-46ad-a7a0-06e0461502f2-openshift-service-ca\") pod \"05054f99-c556-46ad-a7a0-06e0461502f2\" (UID: \"05054f99-c556-46ad-a7a0-06e0461502f2\") "
Jul 17 14:36:05.406329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.406305    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05054f99-c556-46ad-a7a0-06e0461502f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05054f99-c556-46ad-a7a0-06e0461502f2" (UID: "05054f99-c556-46ad-a7a0-06e0461502f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:36:05.407790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.407767    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05054f99-c556-46ad-a7a0-06e0461502f2" (UID: "05054f99-c556-46ad-a7a0-06e0461502f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:36:05.407879 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.407783    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05054f99-c556-46ad-a7a0-06e0461502f2-kube-api-access-nskkt" (OuterVolumeSpecName: "kube-api-access-nskkt") pod "05054f99-c556-46ad-a7a0-06e0461502f2" (UID: "05054f99-c556-46ad-a7a0-06e0461502f2"). InnerVolumeSpecName "kube-api-access-nskkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:36:05.506653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.506622    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05054f99-c556-46ad-a7a0-06e0461502f2-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:05.506653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.506647    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nskkt\" (UniqueName: \"kubernetes.io/projected/05054f99-c556-46ad-a7a0-06e0461502f2-kube-api-access-nskkt\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:05.506653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.506657    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05054f99-c556-46ad-a7a0-06e0461502f2-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:05.550163 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.550135    2575 generic.go:356] "Generic (PLEG): container finished" podID="05054f99-c556-46ad-a7a0-06e0461502f2" containerID="cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74" exitCode=0
Jul 17 14:36:05.550299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.550178    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"
Jul 17 14:36:05.550299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.550226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45" event={"ID":"05054f99-c556-46ad-a7a0-06e0461502f2","Type":"ContainerDied","Data":"cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74"}
Jul 17 14:36:05.550299 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.550288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45" event={"ID":"05054f99-c556-46ad-a7a0-06e0461502f2","Type":"ContainerDied","Data":"1e03e02a240d641ec575115acbf8180018ce8cb50b576b6c37ea6f5a2d3989b4"}
Jul 17 14:36:05.550410 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.550315    2575 scope.go:117] "RemoveContainer" containerID="cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74"
Jul 17 14:36:05.558816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.558796    2575 scope.go:117] "RemoveContainer" containerID="cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74"
Jul 17 14:36:05.559063 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:05.559043    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74\": container with ID starting with cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74 not found: ID does not exist" containerID="cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74"
Jul 17 14:36:05.559112 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.559074    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74"} err="failed to get container status \"cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74\": rpc error: code = NotFound desc = could not find container \"cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74\": container with ID starting with cd985b441ee2fa46a08c55bd6336d23c47e62bf51783d52ea2eeabc9cd0eaa74 not found: ID does not exist"
Jul 17 14:36:05.572500 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.572472    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"]
Jul 17 14:36:05.579185 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:05.579163    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9f570-5d59c7d6d-6kq45"]
Jul 17 14:36:06.074415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:06.074385    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05054f99-c556-46ad-a7a0-06e0461502f2" path="/var/lib/kubelet/pods/05054f99-c556-46ad-a7a0-06e0461502f2/volumes"
Jul 17 14:36:09.409383 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409348    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"]
Jul 17 14:36:09.409755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409705    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:09.409755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409716    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:09.409755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409724    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74eca80a-45df-4e76-9ff4-64584697ffa3" containerName="revoke-keys"
Jul 17 14:36:09.409755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409730    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74eca80a-45df-4e76-9ff4-64584697ffa3" containerName="revoke-keys"
Jul 17 14:36:09.409755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409736    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:09.409755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409741    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:09.409755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409756    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05054f99-c556-46ad-a7a0-06e0461502f2" containerName="maas-api"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409763    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="05054f99-c556-46ad-a7a0-06e0461502f2" containerName="maas-api"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409777    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409782    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409834    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="325de1dc-1b0a-49c6-9fc7-0bc91fe49cff" containerName="revoke-keys"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409843    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409849    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409854    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="05054f99-c556-46ad-a7a0-06e0461502f2" containerName="maas-api"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409860    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a15ace2-9e52-4706-91db-80a0a5a9745f" containerName="revoke-keys"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409865    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:09.409996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.409871    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74eca80a-45df-4e76-9ff4-64584697ffa3" containerName="revoke-keys"
Jul 17 14:36:09.414232 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.414215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"
Jul 17 14:36:09.418007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.417988    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:36:09.426359 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.426340    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"]
Jul 17 14:36:09.538705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.538682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7addc7d0-a87c-4fa5-9db7-d9f30983d8bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw\" (UID: \"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"
Jul 17 14:36:09.639803 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.639775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7addc7d0-a87c-4fa5-9db7-d9f30983d8bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw\" (UID: \"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"
Jul 17 14:36:09.640387 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.640370    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7addc7d0-a87c-4fa5-9db7-d9f30983d8bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw\" (UID: \"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"
Jul 17 14:36:09.723698 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.723675    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"
Jul 17 14:36:09.781028 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.780993    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"]
Jul 17 14:36:09.783929 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:09.783898    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9f570-tgcxw"]
Jul 17 14:36:10.062862 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:10.062839    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"]
Jul 17 14:36:10.063706 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:36:10.063680    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7addc7d0_a87c_4fa5_9db7_d9f30983d8bd.slice/crio-ae04afac9247ce5f50c0e635021f3f3f15197e0b2e0784d5efd6f3fec5c4f288 WatchSource:0}: Error finding container ae04afac9247ce5f50c0e635021f3f3f15197e0b2e0784d5efd6f3fec5c4f288: Status 404 returned error can't find the container with id ae04afac9247ce5f50c0e635021f3f3f15197e0b2e0784d5efd6f3fec5c4f288
Jul 17 14:36:10.079723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:10.079697    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74eca80a-45df-4e76-9ff4-64584697ffa3" path="/var/lib/kubelet/pods/74eca80a-45df-4e76-9ff4-64584697ffa3/volumes"
Jul 17 14:36:10.581592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:10.581551    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" event={"ID":"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd","Type":"ContainerStarted","Data":"e1534cd19e7598c74b6eae8d94d23f5b6f3e1e1d309d1a0895fd656a68b7906e"}
Jul 17 14:36:10.581942 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:10.581613    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" event={"ID":"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd","Type":"ContainerStarted","Data":"ae04afac9247ce5f50c0e635021f3f3f15197e0b2e0784d5efd6f3fec5c4f288"}
Jul 17 14:36:10.604834 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:10.604790    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" podStartSLOduration=1.6047767419999999 podStartE2EDuration="1.604776742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:36:10.602530697 +0000 UTC m=+3048.989878137" watchObservedRunningTime="2026-07-17 14:36:10.604776742 +0000 UTC m=+3048.992124189"
Jul 17 14:36:11.587602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:11.587572    2575 generic.go:356] "Generic (PLEG): container finished" podID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerID="e1534cd19e7598c74b6eae8d94d23f5b6f3e1e1d309d1a0895fd656a68b7906e" exitCode=6
Jul 17 14:36:11.588023 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:11.587675    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" event={"ID":"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd","Type":"ContainerDied","Data":"e1534cd19e7598c74b6eae8d94d23f5b6f3e1e1d309d1a0895fd656a68b7906e"}
Jul 17 14:36:11.588023 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:11.587813    2575 scope.go:117] "RemoveContainer" containerID="e1534cd19e7598c74b6eae8d94d23f5b6f3e1e1d309d1a0895fd656a68b7906e"
Jul 17 14:36:12.594848 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:12.594818    2575 generic.go:356] "Generic (PLEG): container finished" podID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerID="157b17a56a609c757b0f05fe9b781ccfaa30c1a5dbab795755a87c04519f5a42" exitCode=6
Jul 17 14:36:12.595240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:12.594899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" event={"ID":"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd","Type":"ContainerDied","Data":"157b17a56a609c757b0f05fe9b781ccfaa30c1a5dbab795755a87c04519f5a42"}
Jul 17 14:36:12.595240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:12.594956    2575 scope.go:117] "RemoveContainer" containerID="e1534cd19e7598c74b6eae8d94d23f5b6f3e1e1d309d1a0895fd656a68b7906e"
Jul 17 14:36:12.595240 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:12.595044    2575 scope.go:117] "RemoveContainer" containerID="157b17a56a609c757b0f05fe9b781ccfaa30c1a5dbab795755a87c04519f5a42"
Jul 17 14:36:12.595240 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:12.595215    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw_odh-ai-gateway-infra(7addc7d0-a87c-4fa5-9db7-d9f30983d8bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd"
Jul 17 14:36:13.601495 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:13.601466    2575 scope.go:117] "RemoveContainer" containerID="157b17a56a609c757b0f05fe9b781ccfaa30c1a5dbab795755a87c04519f5a42"
Jul 17 14:36:13.601838 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:13.601625    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw_odh-ai-gateway-infra(7addc7d0-a87c-4fa5-9db7-d9f30983d8bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd"
Jul 17 14:36:21.191998 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.191907    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"]
Jul 17 14:36:21.193358 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.193332    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:21.193517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.193496    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a67ae3-b412-4bcc-97f8-bb2f718b5825" containerName="revoke-keys"
Jul 17 14:36:21.198953 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.198934    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.201203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.201182    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8b734e-serving-cert\""
Jul 17 14:36:21.206173 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.206146    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"]
Jul 17 14:36:21.339633 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.339598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qvr\" (UniqueName: \"kubernetes.io/projected/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-kube-api-access-l2qvr\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.339803 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.339651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-maas-api-tls\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.339803 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.339705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.440544 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.440516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qvr\" (UniqueName: \"kubernetes.io/projected/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-kube-api-access-l2qvr\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.440744 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.440563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-maas-api-tls\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.440744 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.440589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.441256 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.441238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.442903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.442862    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-maas-api-tls\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.449369 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.449345    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qvr\" (UniqueName: \"kubernetes.io/projected/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-kube-api-access-l2qvr\") pod \"maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.510251 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.510224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:21.669661 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.669633    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"]
Jul 17 14:36:21.671427 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:36:21.671394    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e08659_298c_4ee3_beb9_9f507b9aa3dd.slice/crio-8a2988970dcc06b74a128d8ac6e0947fa7e433f3dc12ba8afc6b39f2b3a6ef45 WatchSource:0}: Error finding container 8a2988970dcc06b74a128d8ac6e0947fa7e433f3dc12ba8afc6b39f2b3a6ef45: Status 404 returned error can't find the container with id 8a2988970dcc06b74a128d8ac6e0947fa7e433f3dc12ba8afc6b39f2b3a6ef45
Jul 17 14:36:21.673102 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:21.673083    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 14:36:22.648107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.648067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt" event={"ID":"f0e08659-298c-4ee3-beb9-9f507b9aa3dd","Type":"ContainerStarted","Data":"fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c"}
Jul 17 14:36:22.648441 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.648134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt" event={"ID":"f0e08659-298c-4ee3-beb9-9f507b9aa3dd","Type":"ContainerStarted","Data":"8a2988970dcc06b74a128d8ac6e0947fa7e433f3dc12ba8afc6b39f2b3a6ef45"}
Jul 17 14:36:22.648441 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.648164    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:22.668480 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.668441    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt" podStartSLOduration=1.291277171 podStartE2EDuration="1.66842696s" totalImagesPullingTime="377.149789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:36:22.666958645 +0000 UTC m=+3061.054306084" watchObservedRunningTime="2026-07-17 14:36:22.66842696 +0000 UTC m=+3061.055774399"
Jul 17 14:36:22.773938 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.773865    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"]
Jul 17 14:36:22.778925 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.778900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"
Jul 17 14:36:22.789365 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.789336    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"]
Jul 17 14:36:22.954340 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:22.954311    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e89a3-551c-48b0-9a54-12baadb337fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx\" (UID: \"9d9e89a3-551c-48b0-9a54-12baadb337fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"
Jul 17 14:36:23.055453 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:23.055377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e89a3-551c-48b0-9a54-12baadb337fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx\" (UID: \"9d9e89a3-551c-48b0-9a54-12baadb337fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"
Jul 17 14:36:23.055930 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:23.055913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e89a3-551c-48b0-9a54-12baadb337fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx\" (UID: \"9d9e89a3-551c-48b0-9a54-12baadb337fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"
Jul 17 14:36:23.090724 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:23.090701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"
Jul 17 14:36:23.422056 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:23.422036    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"]
Jul 17 14:36:23.424009 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:36:23.423984    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9e89a3_551c_48b0_9a54_12baadb337fa.slice/crio-7bced971c37765ddb20ca82710b19b065e6fad87ebceed8a80fac17242861ec8 WatchSource:0}: Error finding container 7bced971c37765ddb20ca82710b19b065e6fad87ebceed8a80fac17242861ec8: Status 404 returned error can't find the container with id 7bced971c37765ddb20ca82710b19b065e6fad87ebceed8a80fac17242861ec8
Jul 17 14:36:23.654216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:23.654190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" event={"ID":"9d9e89a3-551c-48b0-9a54-12baadb337fa","Type":"ContainerStarted","Data":"7bced971c37765ddb20ca82710b19b065e6fad87ebceed8a80fac17242861ec8"}
Jul 17 14:36:24.069648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:24.069605    2575 scope.go:117] "RemoveContainer" containerID="157b17a56a609c757b0f05fe9b781ccfaa30c1a5dbab795755a87c04519f5a42"
Jul 17 14:36:24.668028 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:24.661620    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerID="d71f096e1d38850b25b19112c58ad8fa7d26b77d3024753b3611f3dc412e6061" exitCode=6
Jul 17 14:36:24.668028 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:24.661759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" event={"ID":"9d9e89a3-551c-48b0-9a54-12baadb337fa","Type":"ContainerDied","Data":"d71f096e1d38850b25b19112c58ad8fa7d26b77d3024753b3611f3dc412e6061"}
Jul 17 14:36:24.668028 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:24.662170    2575 scope.go:117] "RemoveContainer" containerID="d71f096e1d38850b25b19112c58ad8fa7d26b77d3024753b3611f3dc412e6061"
Jul 17 14:36:24.670216 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:24.670178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" event={"ID":"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd","Type":"ContainerStarted","Data":"823e6f1934eed6f5605b6fc158eb0f35f4c966ab00638bd63bb8af9e0c09e59c"}
Jul 17 14:36:25.677184 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.677104    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerID="728d2932a0835397f6847818dbe9e5d1c8be01752377cbc330762d543e2ca28f" exitCode=6
Jul 17 14:36:25.677615 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.677191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" event={"ID":"9d9e89a3-551c-48b0-9a54-12baadb337fa","Type":"ContainerDied","Data":"728d2932a0835397f6847818dbe9e5d1c8be01752377cbc330762d543e2ca28f"}
Jul 17 14:36:25.677615 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.677258    2575 scope.go:117] "RemoveContainer" containerID="d71f096e1d38850b25b19112c58ad8fa7d26b77d3024753b3611f3dc412e6061"
Jul 17 14:36:25.677615 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.677392    2575 scope.go:117] "RemoveContainer" containerID="728d2932a0835397f6847818dbe9e5d1c8be01752377cbc330762d543e2ca28f"
Jul 17 14:36:25.677787 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:25.677641    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx_odh-ai-gateway-infra(9d9e89a3-551c-48b0-9a54-12baadb337fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa"
Jul 17 14:36:25.679100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.679079    2575 generic.go:356] "Generic (PLEG): container finished" podID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerID="823e6f1934eed6f5605b6fc158eb0f35f4c966ab00638bd63bb8af9e0c09e59c" exitCode=6
Jul 17 14:36:25.679194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.679107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" event={"ID":"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd","Type":"ContainerDied","Data":"823e6f1934eed6f5605b6fc158eb0f35f4c966ab00638bd63bb8af9e0c09e59c"}
Jul 17 14:36:25.679359 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.679347    2575 scope.go:117] "RemoveContainer" containerID="823e6f1934eed6f5605b6fc158eb0f35f4c966ab00638bd63bb8af9e0c09e59c"
Jul 17 14:36:25.679497 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:25.679484    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw_odh-ai-gateway-infra(7addc7d0-a87c-4fa5-9db7-d9f30983d8bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd"
Jul 17 14:36:25.688196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.688183    2575 scope.go:117] "RemoveContainer" containerID="157b17a56a609c757b0f05fe9b781ccfaa30c1a5dbab795755a87c04519f5a42"
Jul 17 14:36:25.712828 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:25.712802    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"]
Jul 17 14:36:26.686246 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:26.686223    2575 scope.go:117] "RemoveContainer" containerID="728d2932a0835397f6847818dbe9e5d1c8be01752377cbc330762d543e2ca28f"
Jul 17 14:36:26.686686 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:26.686447    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx_odh-ai-gateway-infra(9d9e89a3-551c-48b0-9a54-12baadb337fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa"
Jul 17 14:36:26.820697 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:26.820669    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"
Jul 17 14:36:26.988660 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:26.988626    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7addc7d0-a87c-4fa5-9db7-d9f30983d8bd-maas-api-service-ca\") pod \"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd\" (UID: \"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd\") "
Jul 17 14:36:26.988975 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:26.988954    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7addc7d0-a87c-4fa5-9db7-d9f30983d8bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" (UID: "7addc7d0-a87c-4fa5-9db7-d9f30983d8bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:36:27.089791 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:27.089758    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7addc7d0-a87c-4fa5-9db7-d9f30983d8bd-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:27.694297 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:27.694245    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"
Jul 17 14:36:27.694784 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:27.694305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw" event={"ID":"7addc7d0-a87c-4fa5-9db7-d9f30983d8bd","Type":"ContainerDied","Data":"ae04afac9247ce5f50c0e635021f3f3f15197e0b2e0784d5efd6f3fec5c4f288"}
Jul 17 14:36:27.694784 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:27.694375    2575 scope.go:117] "RemoveContainer" containerID="823e6f1934eed6f5605b6fc158eb0f35f4c966ab00638bd63bb8af9e0c09e59c"
Jul 17 14:36:27.722688 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:27.722665    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"]
Jul 17 14:36:27.732129 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:27.732101    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-n4pdw"]
Jul 17 14:36:28.075023 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:28.074993    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" path="/var/lib/kubelet/pods/7addc7d0-a87c-4fa5-9db7-d9f30983d8bd/volumes"
Jul 17 14:36:28.660488 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:28.660462    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:29.329392 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329363    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"]
Jul 17 14:36:29.329760 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329747    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:29.329804 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329761    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:29.329804 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329774    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:29.329804 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329780    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:29.329899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329854    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:29.329899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329862    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:29.329899 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.329869    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:29.334255 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.334229    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.336925 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.336904    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b55149-serving-cert\""
Jul 17 14:36:29.351020 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.350997    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"]
Jul 17 14:36:29.508615 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.508580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.508615 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.508616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.508842 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.508644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8ph\" (UniqueName: \"kubernetes.io/projected/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-kube-api-access-cx8ph\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.609687 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.609610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.609687 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.609644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.609687 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.609662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8ph\" (UniqueName: \"kubernetes.io/projected/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-kube-api-access-cx8ph\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.609958 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:29.609784    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-serving-cert: secret "maas-api-e2e-api-b-b55149-serving-cert" not found
Jul 17 14:36:29.609958 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:29.609857    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls podName:9d75c7ad-5225-4008-b6fc-aeafd9e58dcf nodeName:}" failed. No retries permitted until 2026-07-17 14:36:30.109837119 +0000 UTC m=+3068.497184537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls") pod "maas-api-e2e-api-b-b55149-84b9544694-xpc49" (UID: "9d75c7ad-5225-4008-b6fc-aeafd9e58dcf") : secret "maas-api-e2e-api-b-b55149-serving-cert" not found
Jul 17 14:36:29.610422 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.610404    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:29.619767 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:29.619740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8ph\" (UniqueName: \"kubernetes.io/projected/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-kube-api-access-cx8ph\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:30.113425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:30.113394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:30.115739 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:30.115717    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls\") pod \"maas-api-e2e-api-b-b55149-84b9544694-xpc49\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:30.244287 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:30.244254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:30.391013 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:30.390988    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"]
Jul 17 14:36:30.392841 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:36:30.392808    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d75c7ad_5225_4008_b6fc_aeafd9e58dcf.slice/crio-0d378e91e243b2c0906ecfa8d2b973bd99f9f751ec063837fc8dfa2decf05b46 WatchSource:0}: Error finding container 0d378e91e243b2c0906ecfa8d2b973bd99f9f751ec063837fc8dfa2decf05b46: Status 404 returned error can't find the container with id 0d378e91e243b2c0906ecfa8d2b973bd99f9f751ec063837fc8dfa2decf05b46
Jul 17 14:36:30.712730 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:30.712695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49" event={"ID":"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf","Type":"ContainerStarted","Data":"0d378e91e243b2c0906ecfa8d2b973bd99f9f751ec063837fc8dfa2decf05b46"}
Jul 17 14:36:31.719799 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:31.719764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49" event={"ID":"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf","Type":"ContainerStarted","Data":"538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84"}
Jul 17 14:36:31.720210 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:31.719821    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:33.642702 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.642650    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49" podStartSLOduration=4.343105973 podStartE2EDuration="4.642632977s" totalImagesPullingTime="299.527004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:36:31.746521334 +0000 UTC m=+3070.133868767" watchObservedRunningTime="2026-07-17 14:36:33.642632977 +0000 UTC m=+3072.029980415"
Jul 17 14:36:33.643566 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.643547    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"]
Jul 17 14:36:33.643929 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.643918    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:33.643967 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.643932    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7addc7d0-a87c-4fa5-9db7-d9f30983d8bd" containerName="revoke-keys"
Jul 17 14:36:33.646942 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.646928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"
Jul 17 14:36:33.658778 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.658751    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"]
Jul 17 14:36:33.742008 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.741984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d0dd-806a-44c5-be95-5194d276beac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl\" (UID: \"dfb9d0dd-806a-44c5-be95-5194d276beac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"
Jul 17 14:36:33.842456 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.842424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d0dd-806a-44c5-be95-5194d276beac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl\" (UID: \"dfb9d0dd-806a-44c5-be95-5194d276beac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"
Jul 17 14:36:33.843003 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.842984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d0dd-806a-44c5-be95-5194d276beac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl\" (UID: \"dfb9d0dd-806a-44c5-be95-5194d276beac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"
Jul 17 14:36:33.956851 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:33.956788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"
Jul 17 14:36:34.099188 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:34.099165    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"]
Jul 17 14:36:34.101113 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:36:34.101080    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb9d0dd_806a_44c5_be95_5194d276beac.slice/crio-ffb7ee35310431dac94d1b4d6eaed57c68fd56330e00d4932deaf36f342978be WatchSource:0}: Error finding container ffb7ee35310431dac94d1b4d6eaed57c68fd56330e00d4932deaf36f342978be: Status 404 returned error can't find the container with id ffb7ee35310431dac94d1b4d6eaed57c68fd56330e00d4932deaf36f342978be
Jul 17 14:36:34.735166 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:34.735135    2575 generic.go:356] "Generic (PLEG): container finished" podID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerID="85f519b0465caf1789addadc27efd4652054b526cf7cc2c3f3fd035c6cf451d0" exitCode=6
Jul 17 14:36:34.735541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:34.735231    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" event={"ID":"dfb9d0dd-806a-44c5-be95-5194d276beac","Type":"ContainerDied","Data":"85f519b0465caf1789addadc27efd4652054b526cf7cc2c3f3fd035c6cf451d0"}
Jul 17 14:36:34.735541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:34.735317    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" event={"ID":"dfb9d0dd-806a-44c5-be95-5194d276beac","Type":"ContainerStarted","Data":"ffb7ee35310431dac94d1b4d6eaed57c68fd56330e00d4932deaf36f342978be"}
Jul 17 14:36:34.735541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:34.735404    2575 scope.go:117] "RemoveContainer" containerID="85f519b0465caf1789addadc27efd4652054b526cf7cc2c3f3fd035c6cf451d0"
Jul 17 14:36:35.742184 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:35.742146    2575 generic.go:356] "Generic (PLEG): container finished" podID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerID="7ac9093ad7d20b9e44df1b3d41029b6e0223940b7e1bfa296183cb92066a7a0c" exitCode=6
Jul 17 14:36:35.742690 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:35.742235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" event={"ID":"dfb9d0dd-806a-44c5-be95-5194d276beac","Type":"ContainerDied","Data":"7ac9093ad7d20b9e44df1b3d41029b6e0223940b7e1bfa296183cb92066a7a0c"}
Jul 17 14:36:35.742690 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:35.742362    2575 scope.go:117] "RemoveContainer" containerID="85f519b0465caf1789addadc27efd4652054b526cf7cc2c3f3fd035c6cf451d0"
Jul 17 14:36:35.742690 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:35.742586    2575 scope.go:117] "RemoveContainer" containerID="7ac9093ad7d20b9e44df1b3d41029b6e0223940b7e1bfa296183cb92066a7a0c"
Jul 17 14:36:35.742827 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:35.742781    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl_odh-ai-gateway-infra(dfb9d0dd-806a-44c5-be95-5194d276beac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac"
Jul 17 14:36:36.748404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:36.748373    2575 scope.go:117] "RemoveContainer" containerID="7ac9093ad7d20b9e44df1b3d41029b6e0223940b7e1bfa296183cb92066a7a0c"
Jul 17 14:36:36.748789 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:36.748550    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl_odh-ai-gateway-infra(dfb9d0dd-806a-44c5-be95-5194d276beac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac"
Jul 17 14:36:37.729619 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:37.729593    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:36:39.069136 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:39.069103    2575 scope.go:117] "RemoveContainer" containerID="728d2932a0835397f6847818dbe9e5d1c8be01752377cbc330762d543e2ca28f"
Jul 17 14:36:39.765633 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:39.765602    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerID="064325e4baf2c9ed8ec9e8d84d4ee2eb7ab21fe215949da3a5b1ec6e876c858e" exitCode=6
Jul 17 14:36:39.765753 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:39.765677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" event={"ID":"9d9e89a3-551c-48b0-9a54-12baadb337fa","Type":"ContainerDied","Data":"064325e4baf2c9ed8ec9e8d84d4ee2eb7ab21fe215949da3a5b1ec6e876c858e"}
Jul 17 14:36:39.765753 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:39.765740    2575 scope.go:117] "RemoveContainer" containerID="728d2932a0835397f6847818dbe9e5d1c8be01752377cbc330762d543e2ca28f"
Jul 17 14:36:39.766035 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:39.766015    2575 scope.go:117] "RemoveContainer" containerID="064325e4baf2c9ed8ec9e8d84d4ee2eb7ab21fe215949da3a5b1ec6e876c858e"
Jul 17 14:36:39.766259 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:39.766236    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx_odh-ai-gateway-infra(9d9e89a3-551c-48b0-9a54-12baadb337fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa"
Jul 17 14:36:40.796829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:40.796796    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"]
Jul 17 14:36:40.927764 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:40.927740    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"
Jul 17 14:36:41.103297 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.103213    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e89a3-551c-48b0-9a54-12baadb337fa-maas-api-service-ca\") pod \"9d9e89a3-551c-48b0-9a54-12baadb337fa\" (UID: \"9d9e89a3-551c-48b0-9a54-12baadb337fa\") "
Jul 17 14:36:41.103584 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.103565    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9e89a3-551c-48b0-9a54-12baadb337fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d9e89a3-551c-48b0-9a54-12baadb337fa" (UID: "9d9e89a3-551c-48b0-9a54-12baadb337fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:36:41.204024 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.204000    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9e89a3-551c-48b0-9a54-12baadb337fa-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:41.678986 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.678950    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"]
Jul 17 14:36:41.679593 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.679576    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:41.679655 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.679597    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:41.679655 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.679612    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:41.679655 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.679620    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:41.679750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.679721    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:41.679750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.679735    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:41.683317 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.683300    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"
Jul 17 14:36:41.692184 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.692159    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"]
Jul 17 14:36:41.779366 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.779332    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"
Jul 17 14:36:41.779535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.779356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx" event={"ID":"9d9e89a3-551c-48b0-9a54-12baadb337fa","Type":"ContainerDied","Data":"7bced971c37765ddb20ca82710b19b065e6fad87ebceed8a80fac17242861ec8"}
Jul 17 14:36:41.779535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.779422    2575 scope.go:117] "RemoveContainer" containerID="064325e4baf2c9ed8ec9e8d84d4ee2eb7ab21fe215949da3a5b1ec6e876c858e"
Jul 17 14:36:41.801504 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.801484    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"]
Jul 17 14:36:41.807377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.807355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a16c72-87cd-4fad-a5a9-0f66c5a011f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8b734e-qvntd\" (UID: \"40a16c72-87cd-4fad-a5a9-0f66c5a011f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"
Jul 17 14:36:41.808150 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.808132    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-nhvtx"]
Jul 17 14:36:41.908142 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.908121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a16c72-87cd-4fad-a5a9-0f66c5a011f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8b734e-qvntd\" (UID: \"40a16c72-87cd-4fad-a5a9-0f66c5a011f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"
Jul 17 14:36:41.908776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.908755    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a16c72-87cd-4fad-a5a9-0f66c5a011f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8b734e-qvntd\" (UID: \"40a16c72-87cd-4fad-a5a9-0f66c5a011f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"
Jul 17 14:36:41.995422 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:41.995393    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"
Jul 17 14:36:42.074773 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:42.074741    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" path="/var/lib/kubelet/pods/9d9e89a3-551c-48b0-9a54-12baadb337fa/volumes"
Jul 17 14:36:42.344685 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:42.344661    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"]
Jul 17 14:36:42.346509 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:36:42.346452    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a16c72_87cd_4fad_a5a9_0f66c5a011f3.slice/crio-70455b93264f2cb5d379646bd40a9084d3013367c25e24b98427e70967bdcfc8 WatchSource:0}: Error finding container 70455b93264f2cb5d379646bd40a9084d3013367c25e24b98427e70967bdcfc8: Status 404 returned error can't find the container with id 70455b93264f2cb5d379646bd40a9084d3013367c25e24b98427e70967bdcfc8
Jul 17 14:36:42.787093 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:42.787059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd" event={"ID":"40a16c72-87cd-4fad-a5a9-0f66c5a011f3","Type":"ContainerStarted","Data":"b5525b96243f805ede343dfe06191873eb07ead69a64887375d727988f7db919"}
Jul 17 14:36:42.787249 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:42.787114    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd" event={"ID":"40a16c72-87cd-4fad-a5a9-0f66c5a011f3","Type":"ContainerStarted","Data":"70455b93264f2cb5d379646bd40a9084d3013367c25e24b98427e70967bdcfc8"}
Jul 17 14:36:42.806711 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:42.806663    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd" podStartSLOduration=1.806649127 podStartE2EDuration="1.806649127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:36:42.805672554 +0000 UTC m=+3081.193019992" watchObservedRunningTime="2026-07-17 14:36:42.806649127 +0000 UTC m=+3081.193996566"
Jul 17 14:36:43.793386 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:43.793353    2575 generic.go:356] "Generic (PLEG): container finished" podID="40a16c72-87cd-4fad-a5a9-0f66c5a011f3" containerID="b5525b96243f805ede343dfe06191873eb07ead69a64887375d727988f7db919" exitCode=0
Jul 17 14:36:43.793564 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:43.793443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd" event={"ID":"40a16c72-87cd-4fad-a5a9-0f66c5a011f3","Type":"ContainerDied","Data":"b5525b96243f805ede343dfe06191873eb07ead69a64887375d727988f7db919"}
Jul 17 14:36:44.932399 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:44.932378    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"
Jul 17 14:36:45.032515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:45.032491    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a16c72-87cd-4fad-a5a9-0f66c5a011f3-maas-api-service-ca\") pod \"40a16c72-87cd-4fad-a5a9-0f66c5a011f3\" (UID: \"40a16c72-87cd-4fad-a5a9-0f66c5a011f3\") "
Jul 17 14:36:45.032828 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:45.032808    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a16c72-87cd-4fad-a5a9-0f66c5a011f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40a16c72-87cd-4fad-a5a9-0f66c5a011f3" (UID: "40a16c72-87cd-4fad-a5a9-0f66c5a011f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:36:45.133906 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:45.133848    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a16c72-87cd-4fad-a5a9-0f66c5a011f3-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:45.805485 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:45.805447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd" event={"ID":"40a16c72-87cd-4fad-a5a9-0f66c5a011f3","Type":"ContainerDied","Data":"70455b93264f2cb5d379646bd40a9084d3013367c25e24b98427e70967bdcfc8"}
Jul 17 14:36:45.805485 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:45.805478    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"
Jul 17 14:36:45.805688 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:45.805518    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70455b93264f2cb5d379646bd40a9084d3013367c25e24b98427e70967bdcfc8"
Jul 17 14:36:51.069661 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:51.069632    2575 scope.go:117] "RemoveContainer" containerID="7ac9093ad7d20b9e44df1b3d41029b6e0223940b7e1bfa296183cb92066a7a0c"
Jul 17 14:36:51.834526 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:51.834492    2575 generic.go:356] "Generic (PLEG): container finished" podID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerID="0a0cc9a07bf92f1fb53cd56c0002c6c6d518e6ee73da00e5d340faa1dad153b4" exitCode=6
Jul 17 14:36:51.834692 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:51.834565    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" event={"ID":"dfb9d0dd-806a-44c5-be95-5194d276beac","Type":"ContainerDied","Data":"0a0cc9a07bf92f1fb53cd56c0002c6c6d518e6ee73da00e5d340faa1dad153b4"}
Jul 17 14:36:51.834692 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:51.834656    2575 scope.go:117] "RemoveContainer" containerID="7ac9093ad7d20b9e44df1b3d41029b6e0223940b7e1bfa296183cb92066a7a0c"
Jul 17 14:36:51.834966 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:51.834946    2575 scope.go:117] "RemoveContainer" containerID="0a0cc9a07bf92f1fb53cd56c0002c6c6d518e6ee73da00e5d340faa1dad153b4"
Jul 17 14:36:51.835192 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:51.835171    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl_odh-ai-gateway-infra(dfb9d0dd-806a-44c5-be95-5194d276beac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac"
Jul 17 14:36:52.866818 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:52.866790    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"]
Jul 17 14:36:53.000826 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.000801    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"
Jul 17 14:36:53.105595 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.105571    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d0dd-806a-44c5-be95-5194d276beac-maas-api-service-ca\") pod \"dfb9d0dd-806a-44c5-be95-5194d276beac\" (UID: \"dfb9d0dd-806a-44c5-be95-5194d276beac\") "
Jul 17 14:36:53.105892 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.105870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb9d0dd-806a-44c5-be95-5194d276beac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfb9d0dd-806a-44c5-be95-5194d276beac" (UID: "dfb9d0dd-806a-44c5-be95-5194d276beac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:36:53.206378 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.206321    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d0dd-806a-44c5-be95-5194d276beac-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:53.846886 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.846861    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"
Jul 17 14:36:53.847079 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.846892    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl" event={"ID":"dfb9d0dd-806a-44c5-be95-5194d276beac","Type":"ContainerDied","Data":"ffb7ee35310431dac94d1b4d6eaed57c68fd56330e00d4932deaf36f342978be"}
Jul 17 14:36:53.847079 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.846941    2575 scope.go:117] "RemoveContainer" containerID="0a0cc9a07bf92f1fb53cd56c0002c6c6d518e6ee73da00e5d340faa1dad153b4"
Jul 17 14:36:53.869959 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.869933    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"]
Jul 17 14:36:53.878069 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:53.878048    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-pq4rl"]
Jul 17 14:36:54.076422 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:54.076397    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" path="/var/lib/kubelet/pods/dfb9d0dd-806a-44c5-be95-5194d276beac/volumes"
Jul 17 14:36:56.834673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:56.834642    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"]
Jul 17 14:36:56.835205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:56.834882    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt" podUID="f0e08659-298c-4ee3-beb9-9f507b9aa3dd" containerName="maas-api" containerID="cri-o://fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c" gracePeriod=30
Jul 17 14:36:57.105203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.105180    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:57.137903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.137880    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-openshift-service-ca\") pod \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") "
Jul 17 14:36:57.138054 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.137939    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2qvr\" (UniqueName: \"kubernetes.io/projected/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-kube-api-access-l2qvr\") pod \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") "
Jul 17 14:36:57.138158 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.138136    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-maas-api-tls\") pod \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\" (UID: \"f0e08659-298c-4ee3-beb9-9f507b9aa3dd\") "
Jul 17 14:36:57.138325 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.138303    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0e08659-298c-4ee3-beb9-9f507b9aa3dd" (UID: "f0e08659-298c-4ee3-beb9-9f507b9aa3dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:36:57.138651 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.138630    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:57.140176 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.140151    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-kube-api-access-l2qvr" (OuterVolumeSpecName: "kube-api-access-l2qvr") pod "f0e08659-298c-4ee3-beb9-9f507b9aa3dd" (UID: "f0e08659-298c-4ee3-beb9-9f507b9aa3dd"). InnerVolumeSpecName "kube-api-access-l2qvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:36:57.140261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.140217    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0e08659-298c-4ee3-beb9-9f507b9aa3dd" (UID: "f0e08659-298c-4ee3-beb9-9f507b9aa3dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:36:57.240069 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.240046    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qvr\" (UniqueName: \"kubernetes.io/projected/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-kube-api-access-l2qvr\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:57.240069 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.240068    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e08659-298c-4ee3-beb9-9f507b9aa3dd-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:36:57.868851 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.868817    2575 generic.go:356] "Generic (PLEG): container finished" podID="f0e08659-298c-4ee3-beb9-9f507b9aa3dd" containerID="fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c" exitCode=0
Jul 17 14:36:57.869246 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.868869    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"
Jul 17 14:36:57.869246 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.868902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt" event={"ID":"f0e08659-298c-4ee3-beb9-9f507b9aa3dd","Type":"ContainerDied","Data":"fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c"}
Jul 17 14:36:57.869246 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.868974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt" event={"ID":"f0e08659-298c-4ee3-beb9-9f507b9aa3dd","Type":"ContainerDied","Data":"8a2988970dcc06b74a128d8ac6e0947fa7e433f3dc12ba8afc6b39f2b3a6ef45"}
Jul 17 14:36:57.869246 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.868994    2575 scope.go:117] "RemoveContainer" containerID="fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c"
Jul 17 14:36:57.877882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.877864    2575 scope.go:117] "RemoveContainer" containerID="fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c"
Jul 17 14:36:57.878121 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:36:57.878103    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c\": container with ID starting with fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c not found: ID does not exist" containerID="fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c"
Jul 17 14:36:57.878195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.878128    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c"} err="failed to get container status \"fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c\": rpc error: code = NotFound desc = could not find container \"fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c\": container with ID starting with fd2059c323ea0f643718199dbb15a0932d32a7f1936249d510871eac4322cf1c not found: ID does not exist"
Jul 17 14:36:57.890696 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.890672    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"]
Jul 17 14:36:57.894605 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:57.894584    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b734e-7d7f976798-6gtnt"]
Jul 17 14:36:58.075156 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:58.075130    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e08659-298c-4ee3-beb9-9f507b9aa3dd" path="/var/lib/kubelet/pods/f0e08659-298c-4ee3-beb9-9f507b9aa3dd/volumes"
Jul 17 14:36:59.619502 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619472    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"]
Jul 17 14:36:59.619864 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619845    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e08659-298c-4ee3-beb9-9f507b9aa3dd" containerName="maas-api"
Jul 17 14:36:59.619864 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619856    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e08659-298c-4ee3-beb9-9f507b9aa3dd" containerName="maas-api"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619866    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619871    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619881    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619888    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619895    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619902    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619909    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a16c72-87cd-4fad-a5a9-0f66c5a011f3" containerName="revoke-keys"
Jul 17 14:36:59.619932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619914    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a16c72-87cd-4fad-a5a9-0f66c5a011f3" containerName="revoke-keys"
Jul 17 14:36:59.620175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619972    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:36:59.620175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619982    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:36:59.620175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619989    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d9e89a3-551c-48b0-9a54-12baadb337fa" containerName="revoke-keys"
Jul 17 14:36:59.620175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.619994    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a16c72-87cd-4fad-a5a9-0f66c5a011f3" containerName="revoke-keys"
Jul 17 14:36:59.620175 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.620000    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e08659-298c-4ee3-beb9-9f507b9aa3dd" containerName="maas-api"
Jul 17 14:36:59.624348 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.624332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"
Jul 17 14:36:59.626875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.626854    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:36:59.639791 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.639765    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"]
Jul 17 14:36:59.659991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.659965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17073ccd-80b6-4ef5-ab02-5954c157f865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9\" (UID: \"17073ccd-80b6-4ef5-ab02-5954c157f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"
Jul 17 14:36:59.760874 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.760849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17073ccd-80b6-4ef5-ab02-5954c157f865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9\" (UID: \"17073ccd-80b6-4ef5-ab02-5954c157f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"
Jul 17 14:36:59.761454 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.761435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17073ccd-80b6-4ef5-ab02-5954c157f865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9\" (UID: \"17073ccd-80b6-4ef5-ab02-5954c157f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"
Jul 17 14:36:59.934254 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:36:59.934182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"
Jul 17 14:37:00.064575 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:00.064547    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"]
Jul 17 14:37:00.067025 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:37:00.066996    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17073ccd_80b6_4ef5_ab02_5954c157f865.slice/crio-292b9890dc8a9a384b51ec458d4183dfa30c120f1f6dbcd1e5bb0cc67a36d862 WatchSource:0}: Error finding container 292b9890dc8a9a384b51ec458d4183dfa30c120f1f6dbcd1e5bb0cc67a36d862: Status 404 returned error can't find the container with id 292b9890dc8a9a384b51ec458d4183dfa30c120f1f6dbcd1e5bb0cc67a36d862
Jul 17 14:37:00.885553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:00.885524    2575 generic.go:356] "Generic (PLEG): container finished" podID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerID="fe10f514689f8d3902f58e1786dcae0704a822f8b9ca128a8eb9a092687d8808" exitCode=6
Jul 17 14:37:00.885864 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:00.885616    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" event={"ID":"17073ccd-80b6-4ef5-ab02-5954c157f865","Type":"ContainerDied","Data":"fe10f514689f8d3902f58e1786dcae0704a822f8b9ca128a8eb9a092687d8808"}
Jul 17 14:37:00.885864 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:00.885677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" event={"ID":"17073ccd-80b6-4ef5-ab02-5954c157f865","Type":"ContainerStarted","Data":"292b9890dc8a9a384b51ec458d4183dfa30c120f1f6dbcd1e5bb0cc67a36d862"}
Jul 17 14:37:00.885864 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:00.885780    2575 scope.go:117] "RemoveContainer" containerID="fe10f514689f8d3902f58e1786dcae0704a822f8b9ca128a8eb9a092687d8808"
Jul 17 14:37:01.774192 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:01.774151    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"]
Jul 17 14:37:01.776120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:01.776091    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b734e-qvntd"]
Jul 17 14:37:01.891933 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:01.891904    2575 generic.go:356] "Generic (PLEG): container finished" podID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerID="2a53942ff7aad654dbe7583cd5ede056e06ef8ce778acbdb1dae30a5059533f2" exitCode=6
Jul 17 14:37:01.892351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:01.892000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" event={"ID":"17073ccd-80b6-4ef5-ab02-5954c157f865","Type":"ContainerDied","Data":"2a53942ff7aad654dbe7583cd5ede056e06ef8ce778acbdb1dae30a5059533f2"}
Jul 17 14:37:01.892351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:01.892090    2575 scope.go:117] "RemoveContainer" containerID="fe10f514689f8d3902f58e1786dcae0704a822f8b9ca128a8eb9a092687d8808"
Jul 17 14:37:01.892351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:01.892191    2575 scope.go:117] "RemoveContainer" containerID="2a53942ff7aad654dbe7583cd5ede056e06ef8ce778acbdb1dae30a5059533f2"
Jul 17 14:37:01.892516 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:01.892464    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9_odh-ai-gateway-infra(17073ccd-80b6-4ef5-ab02-5954c157f865)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865"
Jul 17 14:37:02.074940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:02.074857    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40a16c72-87cd-4fad-a5a9-0f66c5a011f3" path="/var/lib/kubelet/pods/40a16c72-87cd-4fad-a5a9-0f66c5a011f3/volumes"
Jul 17 14:37:02.898179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:02.898149    2575 scope.go:117] "RemoveContainer" containerID="2a53942ff7aad654dbe7583cd5ede056e06ef8ce778acbdb1dae30a5059533f2"
Jul 17 14:37:02.898581 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:02.898360    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9_odh-ai-gateway-infra(17073ccd-80b6-4ef5-ab02-5954c157f865)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865"
Jul 17 14:37:09.153111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.150191    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"]
Jul 17 14:37:09.153111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.151092    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:37:09.153111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.151115    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:37:09.153111 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.151362    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb9d0dd-806a-44c5-be95-5194d276beac" containerName="revoke-keys"
Jul 17 14:37:09.155342 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.155319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"
Jul 17 14:37:09.161935 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.161913    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"]
Jul 17 14:37:09.242636 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.242608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8890b8e-bd5a-40c5-a2c9-fdce9db8273a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b55149-h8wm2\" (UID: \"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"
Jul 17 14:37:09.343027 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.343000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8890b8e-bd5a-40c5-a2c9-fdce9db8273a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b55149-h8wm2\" (UID: \"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"
Jul 17 14:37:09.343596 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.343579    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8890b8e-bd5a-40c5-a2c9-fdce9db8273a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b55149-h8wm2\" (UID: \"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"
Jul 17 14:37:09.467008 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.466980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"
Jul 17 14:37:09.608488 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.608450    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"]
Jul 17 14:37:09.612377 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:37:09.612346    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8890b8e_bd5a_40c5_a2c9_fdce9db8273a.slice/crio-8e6ac73a7a5d7289d4eab8ef57bd12124fdb03086c84d692be5d9b145196de00 WatchSource:0}: Error finding container 8e6ac73a7a5d7289d4eab8ef57bd12124fdb03086c84d692be5d9b145196de00: Status 404 returned error can't find the container with id 8e6ac73a7a5d7289d4eab8ef57bd12124fdb03086c84d692be5d9b145196de00
Jul 17 14:37:09.932866 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.932832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2" event={"ID":"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a","Type":"ContainerStarted","Data":"c0006ae37c2e2862ad020430b54f4dedbdee2a4f570dd19ce3a4957936c65fea"}
Jul 17 14:37:09.933015 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.932883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2" event={"ID":"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a","Type":"ContainerStarted","Data":"8e6ac73a7a5d7289d4eab8ef57bd12124fdb03086c84d692be5d9b145196de00"}
Jul 17 14:37:09.951332 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:09.951240    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2" podStartSLOduration=0.951222722 podStartE2EDuration="951.222722ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:37:09.950464793 +0000 UTC m=+3108.337812244" watchObservedRunningTime="2026-07-17 14:37:09.951222722 +0000 UTC m=+3108.338570161"
Jul 17 14:37:10.939600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:10.939567    2575 generic.go:356] "Generic (PLEG): container finished" podID="f8890b8e-bd5a-40c5-a2c9-fdce9db8273a" containerID="c0006ae37c2e2862ad020430b54f4dedbdee2a4f570dd19ce3a4957936c65fea" exitCode=0
Jul 17 14:37:10.939982 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:10.939657    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2" event={"ID":"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a","Type":"ContainerDied","Data":"c0006ae37c2e2862ad020430b54f4dedbdee2a4f570dd19ce3a4957936c65fea"}
Jul 17 14:37:12.073318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.073293    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"
Jul 17 14:37:12.165555 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.165531    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8890b8e-bd5a-40c5-a2c9-fdce9db8273a-maas-api-service-ca\") pod \"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a\" (UID: \"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a\") "
Jul 17 14:37:12.165860 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.165838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8890b8e-bd5a-40c5-a2c9-fdce9db8273a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8890b8e-bd5a-40c5-a2c9-fdce9db8273a" (UID: "f8890b8e-bd5a-40c5-a2c9-fdce9db8273a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:37:12.266607 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.266581    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8890b8e-bd5a-40c5-a2c9-fdce9db8273a-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:12.813832 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.813807    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"]
Jul 17 14:37:12.814200 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.814188    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8890b8e-bd5a-40c5-a2c9-fdce9db8273a" containerName="revoke-keys"
Jul 17 14:37:12.814241 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.814201    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8890b8e-bd5a-40c5-a2c9-fdce9db8273a" containerName="revoke-keys"
Jul 17 14:37:12.814290 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.814263    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8890b8e-bd5a-40c5-a2c9-fdce9db8273a" containerName="revoke-keys"
Jul 17 14:37:12.817127 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.817112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"
Jul 17 14:37:12.828035 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.827930    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"]
Jul 17 14:37:12.872661 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.872639    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52beb4c0-6e06-46cd-aab2-e74e34c8f70c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5\" (UID: \"52beb4c0-6e06-46cd-aab2-e74e34c8f70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"
Jul 17 14:37:12.950162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.950140    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"
Jul 17 14:37:12.950308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.950183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2" event={"ID":"f8890b8e-bd5a-40c5-a2c9-fdce9db8273a","Type":"ContainerDied","Data":"8e6ac73a7a5d7289d4eab8ef57bd12124fdb03086c84d692be5d9b145196de00"}
Jul 17 14:37:12.950308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.950228    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e6ac73a7a5d7289d4eab8ef57bd12124fdb03086c84d692be5d9b145196de00"
Jul 17 14:37:12.973715 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.973694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52beb4c0-6e06-46cd-aab2-e74e34c8f70c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5\" (UID: \"52beb4c0-6e06-46cd-aab2-e74e34c8f70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"
Jul 17 14:37:12.974221 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:12.974205    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52beb4c0-6e06-46cd-aab2-e74e34c8f70c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5\" (UID: \"52beb4c0-6e06-46cd-aab2-e74e34c8f70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"
Jul 17 14:37:13.134861 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:13.134788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"
Jul 17 14:37:13.265227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:13.265203    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"]
Jul 17 14:37:13.267450 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:37:13.267422    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52beb4c0_6e06_46cd_aab2_e74e34c8f70c.slice/crio-c9eb044f295c0d2ac55bf47538aed85d32b67b24819fe947b3da34e7561dd10f WatchSource:0}: Error finding container c9eb044f295c0d2ac55bf47538aed85d32b67b24819fe947b3da34e7561dd10f: Status 404 returned error can't find the container with id c9eb044f295c0d2ac55bf47538aed85d32b67b24819fe947b3da34e7561dd10f
Jul 17 14:37:13.956329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:13.956302    2575 generic.go:356] "Generic (PLEG): container finished" podID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerID="65c9b44074651bd8fe5c2fe29851190d889ef9cc24f123e071055807a4973da8" exitCode=6
Jul 17 14:37:13.956472 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:13.956434    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" event={"ID":"52beb4c0-6e06-46cd-aab2-e74e34c8f70c","Type":"ContainerDied","Data":"65c9b44074651bd8fe5c2fe29851190d889ef9cc24f123e071055807a4973da8"}
Jul 17 14:37:13.956541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:13.956478    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" event={"ID":"52beb4c0-6e06-46cd-aab2-e74e34c8f70c","Type":"ContainerStarted","Data":"c9eb044f295c0d2ac55bf47538aed85d32b67b24819fe947b3da34e7561dd10f"}
Jul 17 14:37:13.956679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:13.956663    2575 scope.go:117] "RemoveContainer" containerID="65c9b44074651bd8fe5c2fe29851190d889ef9cc24f123e071055807a4973da8"
Jul 17 14:37:14.963205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:14.963121    2575 generic.go:356] "Generic (PLEG): container finished" podID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerID="fa0e21917a2c7dc2c138c7b0daf54d6d754f86a32fa8a048934cfc537d2a6f4a" exitCode=6
Jul 17 14:37:14.963205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:14.963166    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" event={"ID":"52beb4c0-6e06-46cd-aab2-e74e34c8f70c","Type":"ContainerDied","Data":"fa0e21917a2c7dc2c138c7b0daf54d6d754f86a32fa8a048934cfc537d2a6f4a"}
Jul 17 14:37:14.963729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:14.963212    2575 scope.go:117] "RemoveContainer" containerID="65c9b44074651bd8fe5c2fe29851190d889ef9cc24f123e071055807a4973da8"
Jul 17 14:37:14.963729 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:14.963460    2575 scope.go:117] "RemoveContainer" containerID="fa0e21917a2c7dc2c138c7b0daf54d6d754f86a32fa8a048934cfc537d2a6f4a"
Jul 17 14:37:14.963729 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:14.963687    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5_odh-ai-gateway-infra(52beb4c0-6e06-46cd-aab2-e74e34c8f70c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c"
Jul 17 14:37:15.970218 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:15.970192    2575 scope.go:117] "RemoveContainer" containerID="fa0e21917a2c7dc2c138c7b0daf54d6d754f86a32fa8a048934cfc537d2a6f4a"
Jul 17 14:37:15.970590 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:15.970398    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5_odh-ai-gateway-infra(52beb4c0-6e06-46cd-aab2-e74e34c8f70c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c"
Jul 17 14:37:18.069292 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:18.069246    2575 scope.go:117] "RemoveContainer" containerID="2a53942ff7aad654dbe7583cd5ede056e06ef8ce778acbdb1dae30a5059533f2"
Jul 17 14:37:18.987524 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:18.987492    2575 generic.go:356] "Generic (PLEG): container finished" podID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerID="66ed8f3b46422c393c7aeb34b6a1c83b10b1787eed58c497b21565e7cf8d1199" exitCode=6
Jul 17 14:37:18.987681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:18.987563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" event={"ID":"17073ccd-80b6-4ef5-ab02-5954c157f865","Type":"ContainerDied","Data":"66ed8f3b46422c393c7aeb34b6a1c83b10b1787eed58c497b21565e7cf8d1199"}
Jul 17 14:37:18.987681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:18.987645    2575 scope.go:117] "RemoveContainer" containerID="2a53942ff7aad654dbe7583cd5ede056e06ef8ce778acbdb1dae30a5059533f2"
Jul 17 14:37:18.987919 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:18.987904    2575 scope.go:117] "RemoveContainer" containerID="66ed8f3b46422c393c7aeb34b6a1c83b10b1787eed58c497b21565e7cf8d1199"
Jul 17 14:37:18.988131 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:18.988108    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9_odh-ai-gateway-infra(17073ccd-80b6-4ef5-ab02-5954c157f865)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865"
Jul 17 14:37:20.018643 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:20.018610    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"]
Jul 17 14:37:20.153107 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:20.153086    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"
Jul 17 14:37:20.235933 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:20.235908    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17073ccd-80b6-4ef5-ab02-5954c157f865-maas-api-service-ca\") pod \"17073ccd-80b6-4ef5-ab02-5954c157f865\" (UID: \"17073ccd-80b6-4ef5-ab02-5954c157f865\") "
Jul 17 14:37:20.236273 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:20.236251    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17073ccd-80b6-4ef5-ab02-5954c157f865-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17073ccd-80b6-4ef5-ab02-5954c157f865" (UID: "17073ccd-80b6-4ef5-ab02-5954c157f865"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:37:20.336800 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:20.336720    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17073ccd-80b6-4ef5-ab02-5954c157f865-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:21.001699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:21.001672    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"
Jul 17 14:37:21.001859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:21.001706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9" event={"ID":"17073ccd-80b6-4ef5-ab02-5954c157f865","Type":"ContainerDied","Data":"292b9890dc8a9a384b51ec458d4183dfa30c120f1f6dbcd1e5bb0cc67a36d862"}
Jul 17 14:37:21.001859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:21.001769    2575 scope.go:117] "RemoveContainer" containerID="66ed8f3b46422c393c7aeb34b6a1c83b10b1787eed58c497b21565e7cf8d1199"
Jul 17 14:37:21.023884 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:21.023862    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"]
Jul 17 14:37:21.029536 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:21.029514    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9fgw9"]
Jul 17 14:37:22.074425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.074389    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" path="/var/lib/kubelet/pods/17073ccd-80b6-4ef5-ab02-5954c157f865/volumes"
Jul 17 14:37:22.083538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.083510    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"]
Jul 17 14:37:22.083903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.083887    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:22.083963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.083907    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:22.083963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.083956    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:22.084067 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.083965    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:22.084067 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.084037    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:22.084067 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.084051    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:22.088423 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.088408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"
Jul 17 14:37:22.095738 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.095591    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"]
Jul 17 14:37:22.251535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.251503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f8e3c-a4af-41d7-b416-afb627a1a7e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-flslf\" (UID: \"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"
Jul 17 14:37:22.352898 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.352817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f8e3c-a4af-41d7-b416-afb627a1a7e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-flslf\" (UID: \"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"
Jul 17 14:37:22.353461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.353444    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f8e3c-a4af-41d7-b416-afb627a1a7e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-flslf\" (UID: \"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"
Jul 17 14:37:22.400466 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.400431    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"
Jul 17 14:37:22.532542 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:22.532515    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"]
Jul 17 14:37:22.534178 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:37:22.534151    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7f8e3c_a4af_41d7_b416_afb627a1a7e0.slice/crio-d009a0bd5c0aab2aad91d42081b438a97fdedb67d9912fd0fc82ab8fcc5bba0f WatchSource:0}: Error finding container d009a0bd5c0aab2aad91d42081b438a97fdedb67d9912fd0fc82ab8fcc5bba0f: Status 404 returned error can't find the container with id d009a0bd5c0aab2aad91d42081b438a97fdedb67d9912fd0fc82ab8fcc5bba0f
Jul 17 14:37:23.016407 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:23.016373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" event={"ID":"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0","Type":"ContainerStarted","Data":"77f1066f764dbac40907cb1a1c41e11b54f370d231ca86ee75538e65ab1b1e91"}
Jul 17 14:37:23.016550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:23.016458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" event={"ID":"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0","Type":"ContainerStarted","Data":"d009a0bd5c0aab2aad91d42081b438a97fdedb67d9912fd0fc82ab8fcc5bba0f"}
Jul 17 14:37:23.038583 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:23.038502    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" podStartSLOduration=1.038488876 podStartE2EDuration="1.038488876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:37:23.036664291 +0000 UTC m=+3121.424011730" watchObservedRunningTime="2026-07-17 14:37:23.038488876 +0000 UTC m=+3121.425836356"
Jul 17 14:37:24.026196 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:24.026156    2575 generic.go:356] "Generic (PLEG): container finished" podID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerID="77f1066f764dbac40907cb1a1c41e11b54f370d231ca86ee75538e65ab1b1e91" exitCode=6
Jul 17 14:37:24.026665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:24.026205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" event={"ID":"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0","Type":"ContainerDied","Data":"77f1066f764dbac40907cb1a1c41e11b54f370d231ca86ee75538e65ab1b1e91"}
Jul 17 14:37:24.026665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:24.026495    2575 scope.go:117] "RemoveContainer" containerID="77f1066f764dbac40907cb1a1c41e11b54f370d231ca86ee75538e65ab1b1e91"
Jul 17 14:37:24.734996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:24.734963    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"]
Jul 17 14:37:24.735224 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:24.735197    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49" podUID="9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" containerName="maas-api" containerID="cri-o://538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84" gracePeriod=30
Jul 17 14:37:24.976481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:24.976454    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:37:25.032726 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.032692    2575 generic.go:356] "Generic (PLEG): container finished" podID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerID="8a606cc473fdf460c1e52d6b2568eeedf952f86b2769859d9b546e52a05dde21" exitCode=6
Jul 17 14:37:25.033109 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.032774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" event={"ID":"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0","Type":"ContainerDied","Data":"8a606cc473fdf460c1e52d6b2568eeedf952f86b2769859d9b546e52a05dde21"}
Jul 17 14:37:25.033109 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.032830    2575 scope.go:117] "RemoveContainer" containerID="77f1066f764dbac40907cb1a1c41e11b54f370d231ca86ee75538e65ab1b1e91"
Jul 17 14:37:25.033109 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.032973    2575 scope.go:117] "RemoveContainer" containerID="8a606cc473fdf460c1e52d6b2568eeedf952f86b2769859d9b546e52a05dde21"
Jul 17 14:37:25.033261 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:25.033203    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-flslf_odh-ai-gateway-infra(3b7f8e3c-a4af-41d7-b416-afb627a1a7e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0"
Jul 17 14:37:25.034173 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.034142    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" containerID="538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84" exitCode=0
Jul 17 14:37:25.034293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.034172    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49" event={"ID":"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf","Type":"ContainerDied","Data":"538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84"}
Jul 17 14:37:25.034293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.034231    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49" event={"ID":"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf","Type":"ContainerDied","Data":"0d378e91e243b2c0906ecfa8d2b973bd99f9f751ec063837fc8dfa2decf05b46"}
Jul 17 14:37:25.034293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.034181    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"
Jul 17 14:37:25.043708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.043689    2575 scope.go:117] "RemoveContainer" containerID="538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84"
Jul 17 14:37:25.051716 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.051702    2575 scope.go:117] "RemoveContainer" containerID="538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84"
Jul 17 14:37:25.051958 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:25.051940    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84\": container with ID starting with 538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84 not found: ID does not exist" containerID="538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84"
Jul 17 14:37:25.052012 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.051964    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84"} err="failed to get container status \"538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84\": rpc error: code = NotFound desc = could not find container \"538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84\": container with ID starting with 538fd62dda7ede652a26a5cd804ea5d6e393c9e46ab3a050248417afa10e2e84 not found: ID does not exist"
Jul 17 14:37:25.076268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.076246    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls\") pod \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") "
Jul 17 14:37:25.076362 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.076336    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-openshift-service-ca\") pod \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") "
Jul 17 14:37:25.076403 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.076379    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx8ph\" (UniqueName: \"kubernetes.io/projected/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-kube-api-access-cx8ph\") pod \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\" (UID: \"9d75c7ad-5225-4008-b6fc-aeafd9e58dcf\") "
Jul 17 14:37:25.076676 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.076654    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" (UID: "9d75c7ad-5225-4008-b6fc-aeafd9e58dcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:37:25.078219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.078199    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-kube-api-access-cx8ph" (OuterVolumeSpecName: "kube-api-access-cx8ph") pod "9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" (UID: "9d75c7ad-5225-4008-b6fc-aeafd9e58dcf"). InnerVolumeSpecName "kube-api-access-cx8ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:37:25.078309 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.078223    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" (UID: "9d75c7ad-5225-4008-b6fc-aeafd9e58dcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:37:25.177556 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.177509    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:25.177556 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.177529    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx8ph\" (UniqueName: \"kubernetes.io/projected/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-kube-api-access-cx8ph\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:25.177556 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.177538    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:25.358549 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.358522    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"]
Jul 17 14:37:25.362778 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:25.362755    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b55149-84b9544694-xpc49"]
Jul 17 14:37:26.041582 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:26.041546    2575 scope.go:117] "RemoveContainer" containerID="8a606cc473fdf460c1e52d6b2568eeedf952f86b2769859d9b546e52a05dde21"
Jul 17 14:37:26.041986 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:26.041742    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-flslf_odh-ai-gateway-infra(3b7f8e3c-a4af-41d7-b416-afb627a1a7e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0"
Jul 17 14:37:26.074637 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:26.074610    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" path="/var/lib/kubelet/pods/9d75c7ad-5225-4008-b6fc-aeafd9e58dcf/volumes"
Jul 17 14:37:28.074985 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:28.074958    2575 scope.go:117] "RemoveContainer" containerID="fa0e21917a2c7dc2c138c7b0daf54d6d754f86a32fa8a048934cfc537d2a6f4a"
Jul 17 14:37:29.059201 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:29.059175    2575 generic.go:356] "Generic (PLEG): container finished" podID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerID="e3fba2bd1470b10bdf6e5634d5f893fec9eb0fe700a47e14d9bc42fabc636204" exitCode=6
Jul 17 14:37:29.059391 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:29.059250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" event={"ID":"52beb4c0-6e06-46cd-aab2-e74e34c8f70c","Type":"ContainerDied","Data":"e3fba2bd1470b10bdf6e5634d5f893fec9eb0fe700a47e14d9bc42fabc636204"}
Jul 17 14:37:29.059391 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:29.059321    2575 scope.go:117] "RemoveContainer" containerID="fa0e21917a2c7dc2c138c7b0daf54d6d754f86a32fa8a048934cfc537d2a6f4a"
Jul 17 14:37:29.059641 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:29.059626    2575 scope.go:117] "RemoveContainer" containerID="e3fba2bd1470b10bdf6e5634d5f893fec9eb0fe700a47e14d9bc42fabc636204"
Jul 17 14:37:29.059832 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:29.059816    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5_odh-ai-gateway-infra(52beb4c0-6e06-46cd-aab2-e74e34c8f70c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c"
Jul 17 14:37:29.253315 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:29.253268    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"]
Jul 17 14:37:29.257549 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:29.257527    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b55149-h8wm2"]
Jul 17 14:37:30.074105 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:30.074072    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8890b8e-bd5a-40c5-a2c9-fdce9db8273a" path="/var/lib/kubelet/pods/f8890b8e-bd5a-40c5-a2c9-fdce9db8273a/volumes"
Jul 17 14:37:30.094795 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:30.094766    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"]
Jul 17 14:37:30.227494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:30.227468    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"
Jul 17 14:37:30.320516 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:30.320490    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52beb4c0-6e06-46cd-aab2-e74e34c8f70c-maas-api-service-ca\") pod \"52beb4c0-6e06-46cd-aab2-e74e34c8f70c\" (UID: \"52beb4c0-6e06-46cd-aab2-e74e34c8f70c\") "
Jul 17 14:37:30.320859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:30.320775    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52beb4c0-6e06-46cd-aab2-e74e34c8f70c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52beb4c0-6e06-46cd-aab2-e74e34c8f70c" (UID: "52beb4c0-6e06-46cd-aab2-e74e34c8f70c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:37:30.421083 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:30.421020    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52beb4c0-6e06-46cd-aab2-e74e34c8f70c-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:31.071293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:31.071244    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"
Jul 17 14:37:31.071480 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:31.071251    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5" event={"ID":"52beb4c0-6e06-46cd-aab2-e74e34c8f70c","Type":"ContainerDied","Data":"c9eb044f295c0d2ac55bf47538aed85d32b67b24819fe947b3da34e7561dd10f"}
Jul 17 14:37:31.071480 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:31.071352    2575 scope.go:117] "RemoveContainer" containerID="e3fba2bd1470b10bdf6e5634d5f893fec9eb0fe700a47e14d9bc42fabc636204"
Jul 17 14:37:31.093750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:31.093727    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"]
Jul 17 14:37:31.097265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:31.097242    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gbjh5"]
Jul 17 14:37:32.076220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:32.076191    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" path="/var/lib/kubelet/pods/52beb4c0-6e06-46cd-aab2-e74e34c8f70c/volumes"
Jul 17 14:37:33.726064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726035    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"]
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726431    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" containerName="maas-api"
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726442    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" containerName="maas-api"
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726453    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726459    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726469    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726474    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726493    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:33.726493 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726498    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:33.726740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726553    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d75c7ad-5225-4008-b6fc-aeafd9e58dcf" containerName="maas-api"
Jul 17 14:37:33.726740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726559    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:33.726740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726567    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:33.726740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726573    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:33.726740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.726579    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="17073ccd-80b6-4ef5-ab02-5954c157f865" containerName="revoke-keys"
Jul 17 14:37:33.729643 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.729625    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"
Jul 17 14:37:33.741910 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.741885    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"]
Jul 17 14:37:33.850554 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.850519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd63621-82b7-4daf-9f83-251881de0b79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-khfn9\" (UID: \"8bd63621-82b7-4daf-9f83-251881de0b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"
Jul 17 14:37:33.951611 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.951578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd63621-82b7-4daf-9f83-251881de0b79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-khfn9\" (UID: \"8bd63621-82b7-4daf-9f83-251881de0b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"
Jul 17 14:37:33.952197 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:33.952177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd63621-82b7-4daf-9f83-251881de0b79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-khfn9\" (UID: \"8bd63621-82b7-4daf-9f83-251881de0b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"
Jul 17 14:37:34.038820 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:34.038723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"
Jul 17 14:37:34.175519 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:34.175494    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"]
Jul 17 14:37:34.178153 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:37:34.178116    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd63621_82b7_4daf_9f83_251881de0b79.slice/crio-65b22ed3c4c71608e86cee7e9afdb032e6b9d9151fdda085e7dec52a29105087 WatchSource:0}: Error finding container 65b22ed3c4c71608e86cee7e9afdb032e6b9d9151fdda085e7dec52a29105087: Status 404 returned error can't find the container with id 65b22ed3c4c71608e86cee7e9afdb032e6b9d9151fdda085e7dec52a29105087
Jul 17 14:37:35.095318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:35.095214    2575 generic.go:356] "Generic (PLEG): container finished" podID="8bd63621-82b7-4daf-9f83-251881de0b79" containerID="b4b4eff6e8371d476e99def9cafc438de509464772b00d6af39c274e8c2b153c" exitCode=6
Jul 17 14:37:35.095318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:35.095307    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" event={"ID":"8bd63621-82b7-4daf-9f83-251881de0b79","Type":"ContainerDied","Data":"b4b4eff6e8371d476e99def9cafc438de509464772b00d6af39c274e8c2b153c"}
Jul 17 14:37:35.095737 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:35.095347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" event={"ID":"8bd63621-82b7-4daf-9f83-251881de0b79","Type":"ContainerStarted","Data":"65b22ed3c4c71608e86cee7e9afdb032e6b9d9151fdda085e7dec52a29105087"}
Jul 17 14:37:35.095737 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:35.095395    2575 scope.go:117] "RemoveContainer" containerID="b4b4eff6e8371d476e99def9cafc438de509464772b00d6af39c274e8c2b153c"
Jul 17 14:37:36.101734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:36.101657    2575 generic.go:356] "Generic (PLEG): container finished" podID="8bd63621-82b7-4daf-9f83-251881de0b79" containerID="459ede854172213d63b64a29d89aec502e723de632c86863af9cbed88df73542" exitCode=6
Jul 17 14:37:36.101734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:36.101719    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" event={"ID":"8bd63621-82b7-4daf-9f83-251881de0b79","Type":"ContainerDied","Data":"459ede854172213d63b64a29d89aec502e723de632c86863af9cbed88df73542"}
Jul 17 14:37:36.102171 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:36.101763    2575 scope.go:117] "RemoveContainer" containerID="b4b4eff6e8371d476e99def9cafc438de509464772b00d6af39c274e8c2b153c"
Jul 17 14:37:36.102171 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:36.101932    2575 scope.go:117] "RemoveContainer" containerID="459ede854172213d63b64a29d89aec502e723de632c86863af9cbed88df73542"
Jul 17 14:37:36.102171 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:36.102112    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-khfn9_odh-ai-gateway-infra(8bd63621-82b7-4daf-9f83-251881de0b79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" podUID="8bd63621-82b7-4daf-9f83-251881de0b79"
Jul 17 14:37:37.107891 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:37.107863    2575 scope.go:117] "RemoveContainer" containerID="459ede854172213d63b64a29d89aec502e723de632c86863af9cbed88df73542"
Jul 17 14:37:37.108259 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:37.108016    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-khfn9_odh-ai-gateway-infra(8bd63621-82b7-4daf-9f83-251881de0b79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" podUID="8bd63621-82b7-4daf-9f83-251881de0b79"
Jul 17 14:37:39.069648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:39.069621    2575 scope.go:117] "RemoveContainer" containerID="8a606cc473fdf460c1e52d6b2568eeedf952f86b2769859d9b546e52a05dde21"
Jul 17 14:37:40.123246 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:40.123213    2575 generic.go:356] "Generic (PLEG): container finished" podID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerID="9e881eb91c495ad2ffc368a3942d58fd2547abe33ba4e788b6994111ead5abe4" exitCode=6
Jul 17 14:37:40.123641 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:40.123259    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" event={"ID":"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0","Type":"ContainerDied","Data":"9e881eb91c495ad2ffc368a3942d58fd2547abe33ba4e788b6994111ead5abe4"}
Jul 17 14:37:40.123641 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:40.123345    2575 scope.go:117] "RemoveContainer" containerID="8a606cc473fdf460c1e52d6b2568eeedf952f86b2769859d9b546e52a05dde21"
Jul 17 14:37:40.123641 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:40.123567    2575 scope.go:117] "RemoveContainer" containerID="9e881eb91c495ad2ffc368a3942d58fd2547abe33ba4e788b6994111ead5abe4"
Jul 17 14:37:40.123773 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:40.123746    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-flslf_odh-ai-gateway-infra(3b7f8e3c-a4af-41d7-b416-afb627a1a7e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0"
Jul 17 14:37:41.173306 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.173259    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"]
Jul 17 14:37:41.311117 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.311092    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"
Jul 17 14:37:41.414039 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.414008    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f8e3c-a4af-41d7-b416-afb627a1a7e0-maas-api-service-ca\") pod \"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0\" (UID: \"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0\") "
Jul 17 14:37:41.414434 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.414408    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7f8e3c-a4af-41d7-b416-afb627a1a7e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" (UID: "3b7f8e3c-a4af-41d7-b416-afb627a1a7e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:37:41.514591 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.514558    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f8e3c-a4af-41d7-b416-afb627a1a7e0-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:41.825529 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825439    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"]
Jul 17 14:37:41.825905 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825887    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.825994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825908    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.825994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825933    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.825994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825941    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.825994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825959    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:41.825994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825969    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52beb4c0-6e06-46cd-aab2-e74e34c8f70c" containerName="revoke-keys"
Jul 17 14:37:41.825994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825987    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.825994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.825995    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.826368 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.826090    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.826368 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.826103    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:41.829212 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.829192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:41.831512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.831496    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-244a73-serving-cert\""
Jul 17 14:37:41.845248 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.845220    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"]
Jul 17 14:37:41.916975 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.916946    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:41.917162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.916986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3bde99-f970-40ea-96c9-1e72ca54b482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:41.917162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:41.917101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzl4\" (UniqueName: \"kubernetes.io/projected/5c3bde99-f970-40ea-96c9-1e72ca54b482-kube-api-access-mqzl4\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.017717 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.017680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzl4\" (UniqueName: \"kubernetes.io/projected/5c3bde99-f970-40ea-96c9-1e72ca54b482-kube-api-access-mqzl4\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.017905 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.017746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.017905 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.017769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3bde99-f970-40ea-96c9-1e72ca54b482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.018044 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:42.017899    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-serving-cert: secret "maas-api-e2e-shared-a-244a73-serving-cert" not found
Jul 17 14:37:42.018044 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:42.017966    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls podName:5c3bde99-f970-40ea-96c9-1e72ca54b482 nodeName:}" failed. No retries permitted until 2026-07-17 14:37:42.517945479 +0000 UTC m=+3140.905292898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls") pod "maas-api-e2e-shared-a-244a73-6675584657-vlft7" (UID: "5c3bde99-f970-40ea-96c9-1e72ca54b482") : secret "maas-api-e2e-shared-a-244a73-serving-cert" not found
Jul 17 14:37:42.018404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.018388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3bde99-f970-40ea-96c9-1e72ca54b482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.058102 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.058077    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzl4\" (UniqueName: \"kubernetes.io/projected/5c3bde99-f970-40ea-96c9-1e72ca54b482-kube-api-access-mqzl4\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.136017 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.135990    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"
Jul 17 14:37:42.136156 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.135991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf" event={"ID":"3b7f8e3c-a4af-41d7-b416-afb627a1a7e0","Type":"ContainerDied","Data":"d009a0bd5c0aab2aad91d42081b438a97fdedb67d9912fd0fc82ab8fcc5bba0f"}
Jul 17 14:37:42.136156 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.136127    2575 scope.go:117] "RemoveContainer" containerID="9e881eb91c495ad2ffc368a3942d58fd2547abe33ba4e788b6994111ead5abe4"
Jul 17 14:37:42.185461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.185425    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"]
Jul 17 14:37:42.195041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.195011    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-flslf"]
Jul 17 14:37:42.521941 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.521914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.524311 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.524272    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls\") pod \"maas-api-e2e-shared-a-244a73-6675584657-vlft7\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.740088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.740057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:42.884461 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:42.884427    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"]
Jul 17 14:37:42.886640 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:37:42.886615    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3bde99_f970_40ea_96c9_1e72ca54b482.slice/crio-85adec5b0ba632483caf0565e39f8769538f0c5675cef3edfb46a8ae76310cbb WatchSource:0}: Error finding container 85adec5b0ba632483caf0565e39f8769538f0c5675cef3edfb46a8ae76310cbb: Status 404 returned error can't find the container with id 85adec5b0ba632483caf0565e39f8769538f0c5675cef3edfb46a8ae76310cbb
Jul 17 14:37:43.142777 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:43.142700    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7" event={"ID":"5c3bde99-f970-40ea-96c9-1e72ca54b482","Type":"ContainerStarted","Data":"85adec5b0ba632483caf0565e39f8769538f0c5675cef3edfb46a8ae76310cbb"}
Jul 17 14:37:44.075247 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:44.075219    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" path="/var/lib/kubelet/pods/3b7f8e3c-a4af-41d7-b416-afb627a1a7e0/volumes"
Jul 17 14:37:44.149106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:44.149076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7" event={"ID":"5c3bde99-f970-40ea-96c9-1e72ca54b482","Type":"ContainerStarted","Data":"c64bfedaf611683c3c9490d9bf90d6aa66d79c1f39d8a0b51515ae6ea97849d1"}
Jul 17 14:37:44.149304 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:44.149164    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:44.177150 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:44.177097    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7" podStartSLOduration=2.800143932 podStartE2EDuration="3.177080859s" totalImagesPullingTime="376.936927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:37:44.174830464 +0000 UTC m=+3142.562177906" watchObservedRunningTime="2026-07-17 14:37:44.177080859 +0000 UTC m=+3142.564428299"
Jul 17 14:37:49.855377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.855345    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"]
Jul 17 14:37:49.856522 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.856499    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7f8e3c-a4af-41d7-b416-afb627a1a7e0" containerName="revoke-keys"
Jul 17 14:37:49.861330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.860573    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:49.866333 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.866046    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-9eb655-serving-cert\""
Jul 17 14:37:49.876055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.876028    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"]
Jul 17 14:37:49.887303 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.887255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:49.887421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.887325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:49.887421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.887405    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrhc\" (UniqueName: \"kubernetes.io/projected/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-kube-api-access-dbrhc\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:49.988417 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.988391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:49.988576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.988423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:49.988576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.988467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrhc\" (UniqueName: \"kubernetes.io/projected/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-kube-api-access-dbrhc\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:49.988576 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:49.988551    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-serving-cert: secret "maas-api-e2e-shared-b-9eb655-serving-cert" not found
Jul 17 14:37:49.988709 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:49.988616    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls podName:a18c1d0e-e5c8-407e-9582-6e0bac26fbb1 nodeName:}" failed. No retries permitted until 2026-07-17 14:37:50.488597256 +0000 UTC m=+3148.875944690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls") pod "maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646" (UID: "a18c1d0e-e5c8-407e-9582-6e0bac26fbb1") : secret "maas-api-e2e-shared-b-9eb655-serving-cert" not found
Jul 17 14:37:49.989038 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:49.989018    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:50.002591 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:50.002565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrhc\" (UniqueName: \"kubernetes.io/projected/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-kube-api-access-dbrhc\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:50.159377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:50.159319    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:37:50.493341 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:50.493301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:50.495543 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:50.495520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:50.777213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:50.777123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:50.932038 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:50.932009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"]
Jul 17 14:37:50.933714 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:37:50.933690    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18c1d0e_e5c8_407e_9582_6e0bac26fbb1.slice/crio-5bdef7f707656b573cdc1e638a479c25321c8f6b753c2ac30a21d5ffd786eee8 WatchSource:0}: Error finding container 5bdef7f707656b573cdc1e638a479c25321c8f6b753c2ac30a21d5ffd786eee8: Status 404 returned error can't find the container with id 5bdef7f707656b573cdc1e638a479c25321c8f6b753c2ac30a21d5ffd786eee8
Jul 17 14:37:51.184091 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:51.184000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646" event={"ID":"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1","Type":"ContainerStarted","Data":"5bdef7f707656b573cdc1e638a479c25321c8f6b753c2ac30a21d5ffd786eee8"}
Jul 17 14:37:52.072986 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:52.072957    2575 scope.go:117] "RemoveContainer" containerID="459ede854172213d63b64a29d89aec502e723de632c86863af9cbed88df73542"
Jul 17 14:37:52.190575 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:52.190542    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646" event={"ID":"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1","Type":"ContainerStarted","Data":"cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572"}
Jul 17 14:37:52.190704 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:52.190604    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:52.217038 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:52.216976    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646" podStartSLOduration=2.689913771 podStartE2EDuration="3.216961593s" totalImagesPullingTime="527.047822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:37:52.21591857 +0000 UTC m=+3150.603266012" watchObservedRunningTime="2026-07-17 14:37:52.216961593 +0000 UTC m=+3150.604309062"
Jul 17 14:37:53.197037 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:53.197007    2575 generic.go:356] "Generic (PLEG): container finished" podID="8bd63621-82b7-4daf-9f83-251881de0b79" containerID="ec849397e7721c81c33b1bc9ef316e3772435a4729a0dc5126140c8fba1afe53" exitCode=6
Jul 17 14:37:53.197473 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:53.197075    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" event={"ID":"8bd63621-82b7-4daf-9f83-251881de0b79","Type":"ContainerDied","Data":"ec849397e7721c81c33b1bc9ef316e3772435a4729a0dc5126140c8fba1afe53"}
Jul 17 14:37:53.197473 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:53.197127    2575 scope.go:117] "RemoveContainer" containerID="459ede854172213d63b64a29d89aec502e723de632c86863af9cbed88df73542"
Jul 17 14:37:53.197604 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:53.197498    2575 scope.go:117] "RemoveContainer" containerID="ec849397e7721c81c33b1bc9ef316e3772435a4729a0dc5126140c8fba1afe53"
Jul 17 14:37:53.197706 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:37:53.197664    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-khfn9_odh-ai-gateway-infra(8bd63621-82b7-4daf-9f83-251881de0b79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" podUID="8bd63621-82b7-4daf-9f83-251881de0b79"
Jul 17 14:37:54.255636 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:54.255608    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"]
Jul 17 14:37:54.386383 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:54.386360    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"
Jul 17 14:37:54.428750 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:54.428722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd63621-82b7-4daf-9f83-251881de0b79-maas-api-service-ca\") pod \"8bd63621-82b7-4daf-9f83-251881de0b79\" (UID: \"8bd63621-82b7-4daf-9f83-251881de0b79\") "
Jul 17 14:37:54.429088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:54.429068    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd63621-82b7-4daf-9f83-251881de0b79-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8bd63621-82b7-4daf-9f83-251881de0b79" (UID: "8bd63621-82b7-4daf-9f83-251881de0b79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:37:54.529678 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:54.529619    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd63621-82b7-4daf-9f83-251881de0b79-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:37:55.208982 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:55.208955    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"
Jul 17 14:37:55.209164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:55.209001    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9" event={"ID":"8bd63621-82b7-4daf-9f83-251881de0b79","Type":"ContainerDied","Data":"65b22ed3c4c71608e86cee7e9afdb032e6b9d9151fdda085e7dec52a29105087"}
Jul 17 14:37:55.209164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:55.209055    2575 scope.go:117] "RemoveContainer" containerID="ec849397e7721c81c33b1bc9ef316e3772435a4729a0dc5126140c8fba1afe53"
Jul 17 14:37:55.232679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:55.232654    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"]
Jul 17 14:37:55.237288 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:55.237256    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-khfn9"]
Jul 17 14:37:56.075206 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:56.075177    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" path="/var/lib/kubelet/pods/8bd63621-82b7-4daf-9f83-251881de0b79/volumes"
Jul 17 14:37:58.203475 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:58.203444    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:37:59.674191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674156    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"]
Jul 17 14:37:59.674581 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674559    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674581 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674570    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674581 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674580    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674585    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674599    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674604    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674671    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674679    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.674829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.674686    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd63621-82b7-4daf-9f83-251881de0b79" containerName="revoke-keys"
Jul 17 14:37:59.679494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.679477    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"
Jul 17 14:37:59.682098 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.682078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:37:59.688171 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.688144    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"]
Jul 17 14:37:59.772981 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.772944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9400099-5d22-4401-a77b-729ea7d70c8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9\" (UID: \"d9400099-5d22-4401-a77b-729ea7d70c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"
Jul 17 14:37:59.873962 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.873915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9400099-5d22-4401-a77b-729ea7d70c8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9\" (UID: \"d9400099-5d22-4401-a77b-729ea7d70c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"
Jul 17 14:37:59.874555 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.874536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9400099-5d22-4401-a77b-729ea7d70c8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9\" (UID: \"d9400099-5d22-4401-a77b-729ea7d70c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"
Jul 17 14:37:59.990749 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:37:59.990712    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"
Jul 17 14:38:00.128007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:00.127971    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"]
Jul 17 14:38:00.130272 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:38:00.130247    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9400099_5d22_4401_a77b_729ea7d70c8f.slice/crio-b6615e9ab1e8cfc2b18706772f55819c3a20160cf07550680605bb64068a35fa WatchSource:0}: Error finding container b6615e9ab1e8cfc2b18706772f55819c3a20160cf07550680605bb64068a35fa: Status 404 returned error can't find the container with id b6615e9ab1e8cfc2b18706772f55819c3a20160cf07550680605bb64068a35fa
Jul 17 14:38:00.236861 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:00.236829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" event={"ID":"d9400099-5d22-4401-a77b-729ea7d70c8f","Type":"ContainerStarted","Data":"b6615e9ab1e8cfc2b18706772f55819c3a20160cf07550680605bb64068a35fa"}
Jul 17 14:38:01.243163 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:01.243133    2575 generic.go:356] "Generic (PLEG): container finished" podID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerID="3bd9804c26241579e5fd082ee6887e3f6dc06547940306c3e6a6b7f336e5903a" exitCode=6
Jul 17 14:38:01.243533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:01.243177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" event={"ID":"d9400099-5d22-4401-a77b-729ea7d70c8f","Type":"ContainerDied","Data":"3bd9804c26241579e5fd082ee6887e3f6dc06547940306c3e6a6b7f336e5903a"}
Jul 17 14:38:01.243533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:01.243375    2575 scope.go:117] "RemoveContainer" containerID="3bd9804c26241579e5fd082ee6887e3f6dc06547940306c3e6a6b7f336e5903a"
Jul 17 14:38:02.251076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.251044    2575 generic.go:356] "Generic (PLEG): container finished" podID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerID="7fa688ac3218285868abe3c703998bd83b0f6bb8fd9088d8884d6b0691a7a38a" exitCode=6
Jul 17 14:38:02.251490 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.251127    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" event={"ID":"d9400099-5d22-4401-a77b-729ea7d70c8f","Type":"ContainerDied","Data":"7fa688ac3218285868abe3c703998bd83b0f6bb8fd9088d8884d6b0691a7a38a"}
Jul 17 14:38:02.251490 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.251184    2575 scope.go:117] "RemoveContainer" containerID="3bd9804c26241579e5fd082ee6887e3f6dc06547940306c3e6a6b7f336e5903a"
Jul 17 14:38:02.251490 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.251322    2575 scope.go:117] "RemoveContainer" containerID="7fa688ac3218285868abe3c703998bd83b0f6bb8fd9088d8884d6b0691a7a38a"
Jul 17 14:38:02.251624 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:02.251556    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9_odh-ai-gateway-infra(d9400099-5d22-4401-a77b-729ea7d70c8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f"
Jul 17 14:38:02.862497 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.862466    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"]
Jul 17 14:38:02.867458 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.867436    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"
Jul 17 14:38:02.874064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.873972    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"]
Jul 17 14:38:02.998722 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:02.998689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4296376b-ce3e-4872-ab28-cc982b6ce260-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv\" (UID: \"4296376b-ce3e-4872-ab28-cc982b6ce260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"
Jul 17 14:38:03.099434 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:03.099403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4296376b-ce3e-4872-ab28-cc982b6ce260-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv\" (UID: \"4296376b-ce3e-4872-ab28-cc982b6ce260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"
Jul 17 14:38:03.099932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:03.099911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4296376b-ce3e-4872-ab28-cc982b6ce260-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv\" (UID: \"4296376b-ce3e-4872-ab28-cc982b6ce260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"
Jul 17 14:38:03.179300 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:03.179204    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"
Jul 17 14:38:03.258553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:03.258530    2575 scope.go:117] "RemoveContainer" containerID="7fa688ac3218285868abe3c703998bd83b0f6bb8fd9088d8884d6b0691a7a38a"
Jul 17 14:38:03.258980 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:03.258952    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9_odh-ai-gateway-infra(d9400099-5d22-4401-a77b-729ea7d70c8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f"
Jul 17 14:38:03.325429 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:03.325405    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"]
Jul 17 14:38:03.327488 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:38:03.327460    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4296376b_ce3e_4872_ab28_cc982b6ce260.slice/crio-dbcfb4deef1e5f55439fece54bf3dce724904026934ce39a1b3820b4a8f393ab WatchSource:0}: Error finding container dbcfb4deef1e5f55439fece54bf3dce724904026934ce39a1b3820b4a8f393ab: Status 404 returned error can't find the container with id dbcfb4deef1e5f55439fece54bf3dce724904026934ce39a1b3820b4a8f393ab
Jul 17 14:38:04.264868 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:04.264792    2575 generic.go:356] "Generic (PLEG): container finished" podID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerID="772cb24551a28c98923c9f9ffbebb2de71328a3933bf71b805c69212c43dfe8e" exitCode=6
Jul 17 14:38:04.264868 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:04.264851    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" event={"ID":"4296376b-ce3e-4872-ab28-cc982b6ce260","Type":"ContainerDied","Data":"772cb24551a28c98923c9f9ffbebb2de71328a3933bf71b805c69212c43dfe8e"}
Jul 17 14:38:04.265265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:04.264891    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" event={"ID":"4296376b-ce3e-4872-ab28-cc982b6ce260","Type":"ContainerStarted","Data":"dbcfb4deef1e5f55439fece54bf3dce724904026934ce39a1b3820b4a8f393ab"}
Jul 17 14:38:04.265265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:04.265056    2575 scope.go:117] "RemoveContainer" containerID="772cb24551a28c98923c9f9ffbebb2de71328a3933bf71b805c69212c43dfe8e"
Jul 17 14:38:05.272524 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:05.272494    2575 generic.go:356] "Generic (PLEG): container finished" podID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerID="1249753e57001f3d02df98eb8e40d7767897113efb873302ed211a91114503d5" exitCode=6
Jul 17 14:38:05.272968 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:05.272558    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" event={"ID":"4296376b-ce3e-4872-ab28-cc982b6ce260","Type":"ContainerDied","Data":"1249753e57001f3d02df98eb8e40d7767897113efb873302ed211a91114503d5"}
Jul 17 14:38:05.272968 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:05.272623    2575 scope.go:117] "RemoveContainer" containerID="772cb24551a28c98923c9f9ffbebb2de71328a3933bf71b805c69212c43dfe8e"
Jul 17 14:38:05.272968 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:05.272801    2575 scope.go:117] "RemoveContainer" containerID="1249753e57001f3d02df98eb8e40d7767897113efb873302ed211a91114503d5"
Jul 17 14:38:05.273098 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:05.272991    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv_odh-ai-gateway-infra(4296376b-ce3e-4872-ab28-cc982b6ce260)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260"
Jul 17 14:38:06.279550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:06.279524    2575 scope.go:117] "RemoveContainer" containerID="1249753e57001f3d02df98eb8e40d7767897113efb873302ed211a91114503d5"
Jul 17 14:38:06.279910 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:06.279734    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv_odh-ai-gateway-infra(4296376b-ce3e-4872-ab28-cc982b6ce260)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260"
Jul 17 14:38:15.069512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:15.069482    2575 scope.go:117] "RemoveContainer" containerID="7fa688ac3218285868abe3c703998bd83b0f6bb8fd9088d8884d6b0691a7a38a"
Jul 17 14:38:15.332477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:15.332411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" event={"ID":"d9400099-5d22-4401-a77b-729ea7d70c8f","Type":"ContainerStarted","Data":"2a4fc6db94d28e5fc32cd1f2ee0f392728075c2e1e93ccc4675a380f0e9fd374"}
Jul 17 14:38:15.352747 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:15.352698    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" podStartSLOduration=16.352683738 podStartE2EDuration="16.352683738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:38:15.351553602 +0000 UTC m=+3173.738901042" watchObservedRunningTime="2026-07-17 14:38:15.352683738 +0000 UTC m=+3173.740031176"
Jul 17 14:38:16.338955 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:16.338925    2575 generic.go:356] "Generic (PLEG): container finished" podID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerID="2a4fc6db94d28e5fc32cd1f2ee0f392728075c2e1e93ccc4675a380f0e9fd374" exitCode=6
Jul 17 14:38:16.339393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:16.338997    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" event={"ID":"d9400099-5d22-4401-a77b-729ea7d70c8f","Type":"ContainerDied","Data":"2a4fc6db94d28e5fc32cd1f2ee0f392728075c2e1e93ccc4675a380f0e9fd374"}
Jul 17 14:38:16.339393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:16.339059    2575 scope.go:117] "RemoveContainer" containerID="7fa688ac3218285868abe3c703998bd83b0f6bb8fd9088d8884d6b0691a7a38a"
Jul 17 14:38:16.339393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:16.339376    2575 scope.go:117] "RemoveContainer" containerID="2a4fc6db94d28e5fc32cd1f2ee0f392728075c2e1e93ccc4675a380f0e9fd374"
Jul 17 14:38:16.339592 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:16.339572    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9_odh-ai-gateway-infra(d9400099-5d22-4401-a77b-729ea7d70c8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f"
Jul 17 14:38:16.370700 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:16.370674    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"]
Jul 17 14:38:17.488294 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:17.488259    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"
Jul 17 14:38:17.633788 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:17.633688    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9400099-5d22-4401-a77b-729ea7d70c8f-maas-api-service-ca\") pod \"d9400099-5d22-4401-a77b-729ea7d70c8f\" (UID: \"d9400099-5d22-4401-a77b-729ea7d70c8f\") "
Jul 17 14:38:17.634086 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:17.634063    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9400099-5d22-4401-a77b-729ea7d70c8f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9400099-5d22-4401-a77b-729ea7d70c8f" (UID: "d9400099-5d22-4401-a77b-729ea7d70c8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:38:17.734675 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:17.734636    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9400099-5d22-4401-a77b-729ea7d70c8f-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:38:18.352205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:18.352107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9" event={"ID":"d9400099-5d22-4401-a77b-729ea7d70c8f","Type":"ContainerDied","Data":"b6615e9ab1e8cfc2b18706772f55819c3a20160cf07550680605bb64068a35fa"}
Jul 17 14:38:18.352205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:18.352138    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"
Jul 17 14:38:18.352205 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:18.352174    2575 scope.go:117] "RemoveContainer" containerID="2a4fc6db94d28e5fc32cd1f2ee0f392728075c2e1e93ccc4675a380f0e9fd374"
Jul 17 14:38:18.370055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:18.370029    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"]
Jul 17 14:38:18.377135 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:18.377109    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-hqnx9"]
Jul 17 14:38:19.069105 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:19.069061    2575 scope.go:117] "RemoveContainer" containerID="1249753e57001f3d02df98eb8e40d7767897113efb873302ed211a91114503d5"
Jul 17 14:38:19.359709 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:19.359643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" event={"ID":"4296376b-ce3e-4872-ab28-cc982b6ce260","Type":"ContainerStarted","Data":"fa2c9d27ad79b541317027a0797718b5e3689d50e770573aade6faf7e196a977"}
Jul 17 14:38:19.377252 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:19.377207    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" podStartSLOduration=17.37719502 podStartE2EDuration="17.37719502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:38:19.375370348 +0000 UTC m=+3177.762717788" watchObservedRunningTime="2026-07-17 14:38:19.37719502 +0000 UTC m=+3177.764542459"
Jul 17 14:38:20.075121 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:20.075087    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" path="/var/lib/kubelet/pods/d9400099-5d22-4401-a77b-729ea7d70c8f/volumes"
Jul 17 14:38:20.366924 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:20.366836    2575 generic.go:356] "Generic (PLEG): container finished" podID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerID="fa2c9d27ad79b541317027a0797718b5e3689d50e770573aade6faf7e196a977" exitCode=6
Jul 17 14:38:20.367068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:20.366912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" event={"ID":"4296376b-ce3e-4872-ab28-cc982b6ce260","Type":"ContainerDied","Data":"fa2c9d27ad79b541317027a0797718b5e3689d50e770573aade6faf7e196a977"}
Jul 17 14:38:20.367068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:20.366989    2575 scope.go:117] "RemoveContainer" containerID="1249753e57001f3d02df98eb8e40d7767897113efb873302ed211a91114503d5"
Jul 17 14:38:20.367388 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:20.367371    2575 scope.go:117] "RemoveContainer" containerID="fa2c9d27ad79b541317027a0797718b5e3689d50e770573aade6faf7e196a977"
Jul 17 14:38:20.367636 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:20.367615    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv_odh-ai-gateway-infra(4296376b-ce3e-4872-ab28-cc982b6ce260)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260"
Jul 17 14:38:20.391758 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:20.391727    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"]
Jul 17 14:38:21.503854 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:21.503831    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"
Jul 17 14:38:21.671202 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:21.671119    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4296376b-ce3e-4872-ab28-cc982b6ce260-maas-api-service-ca\") pod \"4296376b-ce3e-4872-ab28-cc982b6ce260\" (UID: \"4296376b-ce3e-4872-ab28-cc982b6ce260\") "
Jul 17 14:38:21.671486 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:21.671465    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4296376b-ce3e-4872-ab28-cc982b6ce260-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4296376b-ce3e-4872-ab28-cc982b6ce260" (UID: "4296376b-ce3e-4872-ab28-cc982b6ce260"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:38:21.772320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:21.772264    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4296376b-ce3e-4872-ab28-cc982b6ce260-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:38:22.153989 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.153951    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"]
Jul 17 14:38:22.154388 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154375    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154389    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154403    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154408    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154415    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154420    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154437    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154443    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154450    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154456    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154461    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154466    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154523    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154531    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154536    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:22.154587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.154543    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:22.157682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.157667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"
Jul 17 14:38:22.166699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.166673    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"]
Jul 17 14:38:22.276167 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.276131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bc7c25-bdec-4356-a3e9-d40bbf9d3725-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-z48bl\" (UID: \"46bc7c25-bdec-4356-a3e9-d40bbf9d3725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"
Jul 17 14:38:22.376708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.376618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bc7c25-bdec-4356-a3e9-d40bbf9d3725-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-z48bl\" (UID: \"46bc7c25-bdec-4356-a3e9-d40bbf9d3725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"
Jul 17 14:38:22.377243 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.377222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bc7c25-bdec-4356-a3e9-d40bbf9d3725-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-z48bl\" (UID: \"46bc7c25-bdec-4356-a3e9-d40bbf9d3725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"
Jul 17 14:38:22.379924 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.379896    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"
Jul 17 14:38:22.380061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.379942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv" event={"ID":"4296376b-ce3e-4872-ab28-cc982b6ce260","Type":"ContainerDied","Data":"dbcfb4deef1e5f55439fece54bf3dce724904026934ce39a1b3820b4a8f393ab"}
Jul 17 14:38:22.380061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.380004    2575 scope.go:117] "RemoveContainer" containerID="fa2c9d27ad79b541317027a0797718b5e3689d50e770573aade6faf7e196a977"
Jul 17 14:38:22.396131 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.396105    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"]
Jul 17 14:38:22.402533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.402504    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-x4vsv"]
Jul 17 14:38:22.469029 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.469000    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"
Jul 17 14:38:22.572993 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.572963    2575 scope.go:117] "RemoveContainer" containerID="3b001632ff92a23df8bcb6940b5dde064ac715b5ecefde942da501c492f8d36f"
Jul 17 14:38:22.602144 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:22.602121    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"]
Jul 17 14:38:22.606656 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:38:22.606627    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bc7c25_bdec_4356_a3e9_d40bbf9d3725.slice/crio-c8e2ce672a551aa62ea6a8f3a036fa20f9b941b6c3fa211bad08392949a7aee0 WatchSource:0}: Error finding container c8e2ce672a551aa62ea6a8f3a036fa20f9b941b6c3fa211bad08392949a7aee0: Status 404 returned error can't find the container with id c8e2ce672a551aa62ea6a8f3a036fa20f9b941b6c3fa211bad08392949a7aee0
Jul 17 14:38:23.386868 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:23.386839    2575 generic.go:356] "Generic (PLEG): container finished" podID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerID="698583c652ef9acba45154f33f003dabfbbfb62c5cb9d56b2d90b1853710e3da" exitCode=6
Jul 17 14:38:23.387082 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:23.386923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" event={"ID":"46bc7c25-bdec-4356-a3e9-d40bbf9d3725","Type":"ContainerDied","Data":"698583c652ef9acba45154f33f003dabfbbfb62c5cb9d56b2d90b1853710e3da"}
Jul 17 14:38:23.387082 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:23.386984    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" event={"ID":"46bc7c25-bdec-4356-a3e9-d40bbf9d3725","Type":"ContainerStarted","Data":"c8e2ce672a551aa62ea6a8f3a036fa20f9b941b6c3fa211bad08392949a7aee0"}
Jul 17 14:38:23.387082 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:23.387060    2575 scope.go:117] "RemoveContainer" containerID="698583c652ef9acba45154f33f003dabfbbfb62c5cb9d56b2d90b1853710e3da"
Jul 17 14:38:24.074893 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:24.074863    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" path="/var/lib/kubelet/pods/4296376b-ce3e-4872-ab28-cc982b6ce260/volumes"
Jul 17 14:38:24.394017 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:24.393983    2575 generic.go:356] "Generic (PLEG): container finished" podID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerID="4a28791c836f1b0e4c08daa4221931b1433c83e26a172d052b66fa23f9566ba2" exitCode=6
Jul 17 14:38:24.394177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:24.394059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" event={"ID":"46bc7c25-bdec-4356-a3e9-d40bbf9d3725","Type":"ContainerDied","Data":"4a28791c836f1b0e4c08daa4221931b1433c83e26a172d052b66fa23f9566ba2"}
Jul 17 14:38:24.394177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:24.394127    2575 scope.go:117] "RemoveContainer" containerID="698583c652ef9acba45154f33f003dabfbbfb62c5cb9d56b2d90b1853710e3da"
Jul 17 14:38:24.394407 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:24.394390    2575 scope.go:117] "RemoveContainer" containerID="4a28791c836f1b0e4c08daa4221931b1433c83e26a172d052b66fa23f9566ba2"
Jul 17 14:38:24.394617 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:24.394599    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-z48bl_odh-ai-gateway-infra(46bc7c25-bdec-4356-a3e9-d40bbf9d3725)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725"
Jul 17 14:38:25.401095 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:25.401066    2575 scope.go:117] "RemoveContainer" containerID="4a28791c836f1b0e4c08daa4221931b1433c83e26a172d052b66fa23f9566ba2"
Jul 17 14:38:25.401699 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:25.401301    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-z48bl_odh-ai-gateway-infra(46bc7c25-bdec-4356-a3e9-d40bbf9d3725)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725"
Jul 17 14:38:33.774774 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.774738    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"]
Jul 17 14:38:33.775192 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.775179    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4296376b-ce3e-4872-ab28-cc982b6ce260" containerName="revoke-keys"
Jul 17 14:38:33.775251 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.775196    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9400099-5d22-4401-a77b-729ea7d70c8f" containerName="revoke-keys"
Jul 17 14:38:33.779753 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.779736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"
Jul 17 14:38:33.788112 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.788085    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"]
Jul 17 14:38:33.870010 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.869971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e5c51-2d19-426e-a7b1-142fda10368d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4\" (UID: \"850e5c51-2d19-426e-a7b1-142fda10368d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"
Jul 17 14:38:33.971009 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.970980    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e5c51-2d19-426e-a7b1-142fda10368d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4\" (UID: \"850e5c51-2d19-426e-a7b1-142fda10368d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"
Jul 17 14:38:33.971572 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:33.971554    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e5c51-2d19-426e-a7b1-142fda10368d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4\" (UID: \"850e5c51-2d19-426e-a7b1-142fda10368d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"
Jul 17 14:38:34.090715 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:34.090645    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"
Jul 17 14:38:34.426832 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:34.426805    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"]
Jul 17 14:38:34.428598 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:38:34.428573    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850e5c51_2d19_426e_a7b1_142fda10368d.slice/crio-46aaf102cd2bac5640fc67fbfaa36f77370ce8ddaec25228b19c480438d0eace WatchSource:0}: Error finding container 46aaf102cd2bac5640fc67fbfaa36f77370ce8ddaec25228b19c480438d0eace: Status 404 returned error can't find the container with id 46aaf102cd2bac5640fc67fbfaa36f77370ce8ddaec25228b19c480438d0eace
Jul 17 14:38:34.447568 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:34.447544    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" event={"ID":"850e5c51-2d19-426e-a7b1-142fda10368d","Type":"ContainerStarted","Data":"46aaf102cd2bac5640fc67fbfaa36f77370ce8ddaec25228b19c480438d0eace"}
Jul 17 14:38:35.455904 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:35.455819    2575 generic.go:356] "Generic (PLEG): container finished" podID="850e5c51-2d19-426e-a7b1-142fda10368d" containerID="02795fd66b1a7bab512577ea7faaa1dae057f96c3b54a7cdd48c958ffd24393a" exitCode=6
Jul 17 14:38:35.456390 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:35.455906    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" event={"ID":"850e5c51-2d19-426e-a7b1-142fda10368d","Type":"ContainerDied","Data":"02795fd66b1a7bab512577ea7faaa1dae057f96c3b54a7cdd48c958ffd24393a"}
Jul 17 14:38:35.456390 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:35.456143    2575 scope.go:117] "RemoveContainer" containerID="02795fd66b1a7bab512577ea7faaa1dae057f96c3b54a7cdd48c958ffd24393a"
Jul 17 14:38:36.462522 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:36.462434    2575 generic.go:356] "Generic (PLEG): container finished" podID="850e5c51-2d19-426e-a7b1-142fda10368d" containerID="87edb7be337324bca20ada53203baefc87ff96f177578b1d1a1bac14406499dc" exitCode=6
Jul 17 14:38:36.462522 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:36.462476    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" event={"ID":"850e5c51-2d19-426e-a7b1-142fda10368d","Type":"ContainerDied","Data":"87edb7be337324bca20ada53203baefc87ff96f177578b1d1a1bac14406499dc"}
Jul 17 14:38:36.462996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:36.462543    2575 scope.go:117] "RemoveContainer" containerID="02795fd66b1a7bab512577ea7faaa1dae057f96c3b54a7cdd48c958ffd24393a"
Jul 17 14:38:36.462996 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:36.462746    2575 scope.go:117] "RemoveContainer" containerID="87edb7be337324bca20ada53203baefc87ff96f177578b1d1a1bac14406499dc"
Jul 17 14:38:36.462996 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:36.462978    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4_odh-ai-gateway-infra(850e5c51-2d19-426e-a7b1-142fda10368d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" podUID="850e5c51-2d19-426e-a7b1-142fda10368d"
Jul 17 14:38:37.471267 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:37.471244    2575 scope.go:117] "RemoveContainer" containerID="87edb7be337324bca20ada53203baefc87ff96f177578b1d1a1bac14406499dc"
Jul 17 14:38:37.471629 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:37.471439    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4_odh-ai-gateway-infra(850e5c51-2d19-426e-a7b1-142fda10368d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" podUID="850e5c51-2d19-426e-a7b1-142fda10368d"
Jul 17 14:38:39.069514 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:39.069487    2575 scope.go:117] "RemoveContainer" containerID="4a28791c836f1b0e4c08daa4221931b1433c83e26a172d052b66fa23f9566ba2"
Jul 17 14:38:39.482340 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:39.482313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" event={"ID":"46bc7c25-bdec-4356-a3e9-d40bbf9d3725","Type":"ContainerStarted","Data":"95af94ddf53afb6d5d46c58d0532e7ffb42d348a5703105d7ae7ab42c518f4d8"}
Jul 17 14:38:39.505863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:39.505822    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" podStartSLOduration=17.505809615 podStartE2EDuration="17.505809615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:38:39.503030056 +0000 UTC m=+3197.890377495" watchObservedRunningTime="2026-07-17 14:38:39.505809615 +0000 UTC m=+3197.893157087"
Jul 17 14:38:40.488994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:40.488956    2575 generic.go:356] "Generic (PLEG): container finished" podID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerID="95af94ddf53afb6d5d46c58d0532e7ffb42d348a5703105d7ae7ab42c518f4d8" exitCode=6
Jul 17 14:38:40.489392 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:40.489034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" event={"ID":"46bc7c25-bdec-4356-a3e9-d40bbf9d3725","Type":"ContainerDied","Data":"95af94ddf53afb6d5d46c58d0532e7ffb42d348a5703105d7ae7ab42c518f4d8"}
Jul 17 14:38:40.489392 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:40.489086    2575 scope.go:117] "RemoveContainer" containerID="4a28791c836f1b0e4c08daa4221931b1433c83e26a172d052b66fa23f9566ba2"
Jul 17 14:38:40.489392 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:40.489378    2575 scope.go:117] "RemoveContainer" containerID="95af94ddf53afb6d5d46c58d0532e7ffb42d348a5703105d7ae7ab42c518f4d8"
Jul 17 14:38:40.489601 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:40.489583    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-z48bl_odh-ai-gateway-infra(46bc7c25-bdec-4356-a3e9-d40bbf9d3725)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725"
Jul 17 14:38:40.528584 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:40.528562    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"]
Jul 17 14:38:41.665267 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:41.665243    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"
Jul 17 14:38:41.738384 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:41.738356    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bc7c25-bdec-4356-a3e9-d40bbf9d3725-maas-api-service-ca\") pod \"46bc7c25-bdec-4356-a3e9-d40bbf9d3725\" (UID: \"46bc7c25-bdec-4356-a3e9-d40bbf9d3725\") "
Jul 17 14:38:41.738671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:41.738650    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bc7c25-bdec-4356-a3e9-d40bbf9d3725-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46bc7c25-bdec-4356-a3e9-d40bbf9d3725" (UID: "46bc7c25-bdec-4356-a3e9-d40bbf9d3725"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:38:41.839776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:41.839716    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bc7c25-bdec-4356-a3e9-d40bbf9d3725-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:38:42.503609 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:42.503578    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"
Jul 17 14:38:42.503779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:42.503614    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl" event={"ID":"46bc7c25-bdec-4356-a3e9-d40bbf9d3725","Type":"ContainerDied","Data":"c8e2ce672a551aa62ea6a8f3a036fa20f9b941b6c3fa211bad08392949a7aee0"}
Jul 17 14:38:42.503779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:42.503667    2575 scope.go:117] "RemoveContainer" containerID="95af94ddf53afb6d5d46c58d0532e7ffb42d348a5703105d7ae7ab42c518f4d8"
Jul 17 14:38:42.521708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:42.521688    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"]
Jul 17 14:38:42.529562 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:42.529545    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-z48bl"]
Jul 17 14:38:44.074523 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:44.074490    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" path="/var/lib/kubelet/pods/46bc7c25-bdec-4356-a3e9-d40bbf9d3725/volumes"
Jul 17 14:38:49.729983 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.729950    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"]
Jul 17 14:38:49.730444 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730378    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730444 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730390    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730444 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730409    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730444 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730414    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730444 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730425    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730444 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730431    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730493    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730500    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.730751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.730506    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46bc7c25-bdec-4356-a3e9-d40bbf9d3725" containerName="revoke-keys"
Jul 17 14:38:49.735090 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.735070    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"
Jul 17 14:38:49.748530 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.748501    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"]
Jul 17 14:38:49.799610 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.799580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b444b00-eaf2-4666-ad98-1470bed29860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4\" (UID: \"1b444b00-eaf2-4666-ad98-1470bed29860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"
Jul 17 14:38:49.900925 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.900895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b444b00-eaf2-4666-ad98-1470bed29860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4\" (UID: \"1b444b00-eaf2-4666-ad98-1470bed29860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"
Jul 17 14:38:49.901477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:49.901460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b444b00-eaf2-4666-ad98-1470bed29860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4\" (UID: \"1b444b00-eaf2-4666-ad98-1470bed29860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"
Jul 17 14:38:50.045409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:50.045347    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"
Jul 17 14:38:50.191561 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:50.191533    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"]
Jul 17 14:38:50.193827 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:38:50.193803    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b444b00_eaf2_4666_ad98_1470bed29860.slice/crio-13c01791fbfe5be8abd8dd1fb971946814f4a0ade9a07f35b3fd3d609d37072e WatchSource:0}: Error finding container 13c01791fbfe5be8abd8dd1fb971946814f4a0ade9a07f35b3fd3d609d37072e: Status 404 returned error can't find the container with id 13c01791fbfe5be8abd8dd1fb971946814f4a0ade9a07f35b3fd3d609d37072e
Jul 17 14:38:50.543122 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:50.543087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" event={"ID":"1b444b00-eaf2-4666-ad98-1470bed29860","Type":"ContainerStarted","Data":"3d5e03426a92b2042f927e787f73de61932458278a41fdf2c7dbed87b0606ec6"}
Jul 17 14:38:50.543330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:50.543142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" event={"ID":"1b444b00-eaf2-4666-ad98-1470bed29860","Type":"ContainerStarted","Data":"13c01791fbfe5be8abd8dd1fb971946814f4a0ade9a07f35b3fd3d609d37072e"}
Jul 17 14:38:50.578408 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:50.578357    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" podStartSLOduration=1.578342577 podStartE2EDuration="1.578342577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:38:50.57784872 +0000 UTC m=+3208.965196160" watchObservedRunningTime="2026-07-17 14:38:50.578342577 +0000 UTC m=+3208.965690016"
Jul 17 14:38:51.549202 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:51.549169    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b444b00-eaf2-4666-ad98-1470bed29860" containerID="3d5e03426a92b2042f927e787f73de61932458278a41fdf2c7dbed87b0606ec6" exitCode=6
Jul 17 14:38:51.549596 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:51.549261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" event={"ID":"1b444b00-eaf2-4666-ad98-1470bed29860","Type":"ContainerDied","Data":"3d5e03426a92b2042f927e787f73de61932458278a41fdf2c7dbed87b0606ec6"}
Jul 17 14:38:51.549596 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:51.549433    2575 scope.go:117] "RemoveContainer" containerID="3d5e03426a92b2042f927e787f73de61932458278a41fdf2c7dbed87b0606ec6"
Jul 17 14:38:52.072667 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.072636    2575 scope.go:117] "RemoveContainer" containerID="87edb7be337324bca20ada53203baefc87ff96f177578b1d1a1bac14406499dc"
Jul 17 14:38:52.555769 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.555740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" event={"ID":"850e5c51-2d19-426e-a7b1-142fda10368d","Type":"ContainerStarted","Data":"2db223fb9d3489ce495e1c9db1039dd2da6449dcee6695ddc0948c98b9125944"}
Jul 17 14:38:52.557478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.557456    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b444b00-eaf2-4666-ad98-1470bed29860" containerID="d9ac34ef8b04d03bd8e65eb8949f6ff2e733e00de8614f0d12b5150facbd7df2" exitCode=6
Jul 17 14:38:52.557567 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.557540    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" event={"ID":"1b444b00-eaf2-4666-ad98-1470bed29860","Type":"ContainerDied","Data":"d9ac34ef8b04d03bd8e65eb8949f6ff2e733e00de8614f0d12b5150facbd7df2"}
Jul 17 14:38:52.557607 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.557590    2575 scope.go:117] "RemoveContainer" containerID="3d5e03426a92b2042f927e787f73de61932458278a41fdf2c7dbed87b0606ec6"
Jul 17 14:38:52.557687 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.557674    2575 scope.go:117] "RemoveContainer" containerID="d9ac34ef8b04d03bd8e65eb8949f6ff2e733e00de8614f0d12b5150facbd7df2"
Jul 17 14:38:52.557848 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:52.557831    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4_odh-ai-gateway-infra(1b444b00-eaf2-4666-ad98-1470bed29860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" podUID="1b444b00-eaf2-4666-ad98-1470bed29860"
Jul 17 14:38:52.579270 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.579231    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" podStartSLOduration=19.579218524 podStartE2EDuration="19.579218524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:38:52.576945742 +0000 UTC m=+3210.964293182" watchObservedRunningTime="2026-07-17 14:38:52.579218524 +0000 UTC m=+3210.966565963"
Jul 17 14:38:52.927999 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.927926    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"]
Jul 17 14:38:52.931504 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.931489    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"
Jul 17 14:38:52.944421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:52.944399    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"]
Jul 17 14:38:53.030658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.030629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10eb41f-52bd-4895-a7de-f160894e10bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd\" (UID: \"a10eb41f-52bd-4895-a7de-f160894e10bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"
Jul 17 14:38:53.131814 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.131783    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10eb41f-52bd-4895-a7de-f160894e10bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd\" (UID: \"a10eb41f-52bd-4895-a7de-f160894e10bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"
Jul 17 14:38:53.132396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.132378    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10eb41f-52bd-4895-a7de-f160894e10bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd\" (UID: \"a10eb41f-52bd-4895-a7de-f160894e10bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"
Jul 17 14:38:53.241222 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.241198    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"
Jul 17 14:38:53.568705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.568631    2575 generic.go:356] "Generic (PLEG): container finished" podID="850e5c51-2d19-426e-a7b1-142fda10368d" containerID="2db223fb9d3489ce495e1c9db1039dd2da6449dcee6695ddc0948c98b9125944" exitCode=6
Jul 17 14:38:53.569100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.568703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" event={"ID":"850e5c51-2d19-426e-a7b1-142fda10368d","Type":"ContainerDied","Data":"2db223fb9d3489ce495e1c9db1039dd2da6449dcee6695ddc0948c98b9125944"}
Jul 17 14:38:53.569100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.568764    2575 scope.go:117] "RemoveContainer" containerID="87edb7be337324bca20ada53203baefc87ff96f177578b1d1a1bac14406499dc"
Jul 17 14:38:53.569100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.569058    2575 scope.go:117] "RemoveContainer" containerID="2db223fb9d3489ce495e1c9db1039dd2da6449dcee6695ddc0948c98b9125944"
Jul 17 14:38:53.569322 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:53.569293    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4_odh-ai-gateway-infra(850e5c51-2d19-426e-a7b1-142fda10368d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" podUID="850e5c51-2d19-426e-a7b1-142fda10368d"
Jul 17 14:38:53.570426 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.570406    2575 scope.go:117] "RemoveContainer" containerID="d9ac34ef8b04d03bd8e65eb8949f6ff2e733e00de8614f0d12b5150facbd7df2"
Jul 17 14:38:53.570631 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:53.570613    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4_odh-ai-gateway-infra(1b444b00-eaf2-4666-ad98-1470bed29860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" podUID="1b444b00-eaf2-4666-ad98-1470bed29860"
Jul 17 14:38:53.602099 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:38:53.602077    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10eb41f_52bd_4895_a7de_f160894e10bb.slice/crio-b539e043c1ce68c7d669d1b0bfc30df1380f385557558efbbd99c85a5ddbbc95 WatchSource:0}: Error finding container b539e043c1ce68c7d669d1b0bfc30df1380f385557558efbbd99c85a5ddbbc95: Status 404 returned error can't find the container with id b539e043c1ce68c7d669d1b0bfc30df1380f385557558efbbd99c85a5ddbbc95
Jul 17 14:38:53.605679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.605656    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"]
Jul 17 14:38:53.607465 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:53.607427    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"]
Jul 17 14:38:54.578136 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.578098    2575 generic.go:356] "Generic (PLEG): container finished" podID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerID="57d7c053cc46ab849c7cd414fdd32b0f7dd8a148b7e8ee25c7fc95df1b0aa8ba" exitCode=6
Jul 17 14:38:54.578576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.578174    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" event={"ID":"a10eb41f-52bd-4895-a7de-f160894e10bb","Type":"ContainerDied","Data":"57d7c053cc46ab849c7cd414fdd32b0f7dd8a148b7e8ee25c7fc95df1b0aa8ba"}
Jul 17 14:38:54.578576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.578228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" event={"ID":"a10eb41f-52bd-4895-a7de-f160894e10bb","Type":"ContainerStarted","Data":"b539e043c1ce68c7d669d1b0bfc30df1380f385557558efbbd99c85a5ddbbc95"}
Jul 17 14:38:54.578576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.578327    2575 scope.go:117] "RemoveContainer" containerID="57d7c053cc46ab849c7cd414fdd32b0f7dd8a148b7e8ee25c7fc95df1b0aa8ba"
Jul 17 14:38:54.737985 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.737953    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"
Jul 17 14:38:54.844988 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.844902    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e5c51-2d19-426e-a7b1-142fda10368d-maas-api-service-ca\") pod \"850e5c51-2d19-426e-a7b1-142fda10368d\" (UID: \"850e5c51-2d19-426e-a7b1-142fda10368d\") "
Jul 17 14:38:54.845393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.845362    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850e5c51-2d19-426e-a7b1-142fda10368d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "850e5c51-2d19-426e-a7b1-142fda10368d" (UID: "850e5c51-2d19-426e-a7b1-142fda10368d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:38:54.945540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:54.945506    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e5c51-2d19-426e-a7b1-142fda10368d-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:38:55.584246 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.584218    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"
Jul 17 14:38:55.584652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.584250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4" event={"ID":"850e5c51-2d19-426e-a7b1-142fda10368d","Type":"ContainerDied","Data":"46aaf102cd2bac5640fc67fbfaa36f77370ce8ddaec25228b19c480438d0eace"}
Jul 17 14:38:55.584652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.584341    2575 scope.go:117] "RemoveContainer" containerID="2db223fb9d3489ce495e1c9db1039dd2da6449dcee6695ddc0948c98b9125944"
Jul 17 14:38:55.586140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.586119    2575 generic.go:356] "Generic (PLEG): container finished" podID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerID="7e8b238d6f92392417bb3d90d5bfd2c7b59378197c3069b58fed2ca12d494095" exitCode=6
Jul 17 14:38:55.586265 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.586160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" event={"ID":"a10eb41f-52bd-4895-a7de-f160894e10bb","Type":"ContainerDied","Data":"7e8b238d6f92392417bb3d90d5bfd2c7b59378197c3069b58fed2ca12d494095"}
Jul 17 14:38:55.586377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.586365    2575 scope.go:117] "RemoveContainer" containerID="7e8b238d6f92392417bb3d90d5bfd2c7b59378197c3069b58fed2ca12d494095"
Jul 17 14:38:55.586543 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:55.586526    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd_odh-ai-gateway-infra(a10eb41f-52bd-4895-a7de-f160894e10bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb"
Jul 17 14:38:55.593980 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.593961    2575 scope.go:117] "RemoveContainer" containerID="57d7c053cc46ab849c7cd414fdd32b0f7dd8a148b7e8ee25c7fc95df1b0aa8ba"
Jul 17 14:38:55.624792 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.624742    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"]
Jul 17 14:38:55.629743 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:55.629722    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-8qhs4"]
Jul 17 14:38:56.074909 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:56.074881    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" path="/var/lib/kubelet/pods/850e5c51-2d19-426e-a7b1-142fda10368d/volumes"
Jul 17 14:38:56.593673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:38:56.593648    2575 scope.go:117] "RemoveContainer" containerID="7e8b238d6f92392417bb3d90d5bfd2c7b59378197c3069b58fed2ca12d494095"
Jul 17 14:38:56.594062 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:38:56.593829    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd_odh-ai-gateway-infra(a10eb41f-52bd-4895-a7de-f160894e10bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb"
Jul 17 14:39:05.069145 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:05.069111    2575 scope.go:117] "RemoveContainer" containerID="d9ac34ef8b04d03bd8e65eb8949f6ff2e733e00de8614f0d12b5150facbd7df2"
Jul 17 14:39:05.642451 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:05.642388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" event={"ID":"1b444b00-eaf2-4666-ad98-1470bed29860","Type":"ContainerStarted","Data":"e14442635d9f39447f6087b6349c8c5b122ad9cbc0ac7da2a5fa99eac6c920c7"}
Jul 17 14:39:06.649261 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:06.649230    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b444b00-eaf2-4666-ad98-1470bed29860" containerID="e14442635d9f39447f6087b6349c8c5b122ad9cbc0ac7da2a5fa99eac6c920c7" exitCode=6
Jul 17 14:39:06.649754 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:06.649307    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" event={"ID":"1b444b00-eaf2-4666-ad98-1470bed29860","Type":"ContainerDied","Data":"e14442635d9f39447f6087b6349c8c5b122ad9cbc0ac7da2a5fa99eac6c920c7"}
Jul 17 14:39:06.649754 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:06.649384    2575 scope.go:117] "RemoveContainer" containerID="d9ac34ef8b04d03bd8e65eb8949f6ff2e733e00de8614f0d12b5150facbd7df2"
Jul 17 14:39:06.649754 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:06.649661    2575 scope.go:117] "RemoveContainer" containerID="e14442635d9f39447f6087b6349c8c5b122ad9cbc0ac7da2a5fa99eac6c920c7"
Jul 17 14:39:06.649894 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:06.649875    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4_odh-ai-gateway-infra(1b444b00-eaf2-4666-ad98-1470bed29860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" podUID="1b444b00-eaf2-4666-ad98-1470bed29860"
Jul 17 14:39:06.675688 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:06.675664    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"]
Jul 17 14:39:07.791908 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:07.791882    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"
Jul 17 14:39:07.850642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:07.850608    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b444b00-eaf2-4666-ad98-1470bed29860-maas-api-service-ca\") pod \"1b444b00-eaf2-4666-ad98-1470bed29860\" (UID: \"1b444b00-eaf2-4666-ad98-1470bed29860\") "
Jul 17 14:39:07.850934 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:07.850914    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b444b00-eaf2-4666-ad98-1470bed29860-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b444b00-eaf2-4666-ad98-1470bed29860" (UID: "1b444b00-eaf2-4666-ad98-1470bed29860"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:39:07.952086 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:07.952004    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b444b00-eaf2-4666-ad98-1470bed29860-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:39:08.661859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:08.661833    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"
Jul 17 14:39:08.662063 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:08.661865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4" event={"ID":"1b444b00-eaf2-4666-ad98-1470bed29860","Type":"ContainerDied","Data":"13c01791fbfe5be8abd8dd1fb971946814f4a0ade9a07f35b3fd3d609d37072e"}
Jul 17 14:39:08.662063 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:08.661918    2575 scope.go:117] "RemoveContainer" containerID="e14442635d9f39447f6087b6349c8c5b122ad9cbc0ac7da2a5fa99eac6c920c7"
Jul 17 14:39:08.684035 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:08.684009    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"]
Jul 17 14:39:08.690660 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:08.690637    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2tlj4"]
Jul 17 14:39:10.074691 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:10.074650    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" path="/var/lib/kubelet/pods/1b444b00-eaf2-4666-ad98-1470bed29860/volumes"
Jul 17 14:39:11.069336 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:11.069296    2575 scope.go:117] "RemoveContainer" containerID="7e8b238d6f92392417bb3d90d5bfd2c7b59378197c3069b58fed2ca12d494095"
Jul 17 14:39:11.679951 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:11.679873    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" event={"ID":"a10eb41f-52bd-4895-a7de-f160894e10bb","Type":"ContainerStarted","Data":"f766250967842420b0fdba7c0dc51d2fd91db614f2385edffde7a35b24532c19"}
Jul 17 14:39:11.703328 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:11.703254    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" podStartSLOduration=19.703237294 podStartE2EDuration="19.703237294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:39:11.703164173 +0000 UTC m=+3230.090511638" watchObservedRunningTime="2026-07-17 14:39:11.703237294 +0000 UTC m=+3230.090584735"
Jul 17 14:39:12.688545 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:12.688511    2575 generic.go:356] "Generic (PLEG): container finished" podID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerID="f766250967842420b0fdba7c0dc51d2fd91db614f2385edffde7a35b24532c19" exitCode=6
Jul 17 14:39:12.688937 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:12.688585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" event={"ID":"a10eb41f-52bd-4895-a7de-f160894e10bb","Type":"ContainerDied","Data":"f766250967842420b0fdba7c0dc51d2fd91db614f2385edffde7a35b24532c19"}
Jul 17 14:39:12.688937 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:12.688656    2575 scope.go:117] "RemoveContainer" containerID="7e8b238d6f92392417bb3d90d5bfd2c7b59378197c3069b58fed2ca12d494095"
Jul 17 14:39:12.689030 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:12.688942    2575 scope.go:117] "RemoveContainer" containerID="f766250967842420b0fdba7c0dc51d2fd91db614f2385edffde7a35b24532c19"
Jul 17 14:39:12.689143 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:12.689122    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd_odh-ai-gateway-infra(a10eb41f-52bd-4895-a7de-f160894e10bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb"
Jul 17 14:39:12.716916 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:12.716890    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"]
Jul 17 14:39:13.828492 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:13.828466    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"
Jul 17 14:39:13.904851 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:13.904817    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10eb41f-52bd-4895-a7de-f160894e10bb-maas-api-service-ca\") pod \"a10eb41f-52bd-4895-a7de-f160894e10bb\" (UID: \"a10eb41f-52bd-4895-a7de-f160894e10bb\") "
Jul 17 14:39:13.905188 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:13.905162    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10eb41f-52bd-4895-a7de-f160894e10bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a10eb41f-52bd-4895-a7de-f160894e10bb" (UID: "a10eb41f-52bd-4895-a7de-f160894e10bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:39:14.006109 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:14.006069    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10eb41f-52bd-4895-a7de-f160894e10bb-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:39:14.701707 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:14.701683    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"
Jul 17 14:39:14.701873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:14.701682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd" event={"ID":"a10eb41f-52bd-4895-a7de-f160894e10bb","Type":"ContainerDied","Data":"b539e043c1ce68c7d669d1b0bfc30df1380f385557558efbbd99c85a5ddbbc95"}
Jul 17 14:39:14.701873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:14.701822    2575 scope.go:117] "RemoveContainer" containerID="f766250967842420b0fdba7c0dc51d2fd91db614f2385edffde7a35b24532c19"
Jul 17 14:39:14.729455 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:14.729419    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"]
Jul 17 14:39:14.736081 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:14.736060    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-gdpcd"]
Jul 17 14:39:16.074473 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:16.074433    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" path="/var/lib/kubelet/pods/a10eb41f-52bd-4895-a7de-f160894e10bb/volumes"
Jul 17 14:39:22.209217 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209183    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"]
Jul 17 14:39:22.209609 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209575    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:22.209609 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209586    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:22.209609 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209600    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.209609 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209608    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209623    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209628    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209634    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209640    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209647    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209651    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209657    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209662    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209671    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209676    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209684    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209690    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209742    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.209748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209748    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.210157 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209754    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:22.210157 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209761    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:22.210157 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209767    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:22.210157 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.209773    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:22.212870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.212854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"
Jul 17 14:39:22.215351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.215326    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:39:22.221932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.221908    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"]
Jul 17 14:39:22.377924 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.377896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c90aa8b-5bab-4228-b259-a4feee902107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-nrmrj\" (UID: \"5c90aa8b-5bab-4228-b259-a4feee902107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"
Jul 17 14:39:22.478823 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.478796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c90aa8b-5bab-4228-b259-a4feee902107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-nrmrj\" (UID: \"5c90aa8b-5bab-4228-b259-a4feee902107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"
Jul 17 14:39:22.481118 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.479393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c90aa8b-5bab-4228-b259-a4feee902107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-nrmrj\" (UID: \"5c90aa8b-5bab-4228-b259-a4feee902107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"
Jul 17 14:39:22.524041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.524019    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"
Jul 17 14:39:22.652607 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.652584    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"]
Jul 17 14:39:22.655104 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:39:22.655078    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c90aa8b_5bab_4228_b259_a4feee902107.slice/crio-5b9ae2115c91bfee18febcff8f0c3c3fad318b4a2c81edb2df22b5fd9a4e9031 WatchSource:0}: Error finding container 5b9ae2115c91bfee18febcff8f0c3c3fad318b4a2c81edb2df22b5fd9a4e9031: Status 404 returned error can't find the container with id 5b9ae2115c91bfee18febcff8f0c3c3fad318b4a2c81edb2df22b5fd9a4e9031
Jul 17 14:39:22.744506 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:22.744452    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" event={"ID":"5c90aa8b-5bab-4228-b259-a4feee902107","Type":"ContainerStarted","Data":"5b9ae2115c91bfee18febcff8f0c3c3fad318b4a2c81edb2df22b5fd9a4e9031"}
Jul 17 14:39:23.750477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:23.750438    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c90aa8b-5bab-4228-b259-a4feee902107" containerID="bf2352c219089009c56a04193ab1329567d6758c07b77fa88458cdab518a4cb4" exitCode=6
Jul 17 14:39:23.750859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:23.750483    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" event={"ID":"5c90aa8b-5bab-4228-b259-a4feee902107","Type":"ContainerDied","Data":"bf2352c219089009c56a04193ab1329567d6758c07b77fa88458cdab518a4cb4"}
Jul 17 14:39:23.750859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:23.750655    2575 scope.go:117] "RemoveContainer" containerID="bf2352c219089009c56a04193ab1329567d6758c07b77fa88458cdab518a4cb4"
Jul 17 14:39:24.757514 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:24.757481    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c90aa8b-5bab-4228-b259-a4feee902107" containerID="1d79f7fcef6ee4db9ae8226ecfd0cbc6368c4e0b244d2463b44d7fb7625ae4c0" exitCode=6
Jul 17 14:39:24.757879 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:24.757559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" event={"ID":"5c90aa8b-5bab-4228-b259-a4feee902107","Type":"ContainerDied","Data":"1d79f7fcef6ee4db9ae8226ecfd0cbc6368c4e0b244d2463b44d7fb7625ae4c0"}
Jul 17 14:39:24.757879 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:24.757614    2575 scope.go:117] "RemoveContainer" containerID="bf2352c219089009c56a04193ab1329567d6758c07b77fa88458cdab518a4cb4"
Jul 17 14:39:24.757879 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:24.757838    2575 scope.go:117] "RemoveContainer" containerID="1d79f7fcef6ee4db9ae8226ecfd0cbc6368c4e0b244d2463b44d7fb7625ae4c0"
Jul 17 14:39:24.758051 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:24.758035    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-nrmrj_odh-ai-gateway-infra(5c90aa8b-5bab-4228-b259-a4feee902107)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" podUID="5c90aa8b-5bab-4228-b259-a4feee902107"
Jul 17 14:39:25.764002 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:25.763976    2575 scope.go:117] "RemoveContainer" containerID="1d79f7fcef6ee4db9ae8226ecfd0cbc6368c4e0b244d2463b44d7fb7625ae4c0"
Jul 17 14:39:25.764378 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:25.764166    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-nrmrj_odh-ai-gateway-infra(5c90aa8b-5bab-4228-b259-a4feee902107)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" podUID="5c90aa8b-5bab-4228-b259-a4feee902107"
Jul 17 14:39:33.824977 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.824943    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"]
Jul 17 14:39:33.825367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.825332    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:33.825367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.825344    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:33.825445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.825417    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="850e5c51-2d19-426e-a7b1-142fda10368d" containerName="revoke-keys"
Jul 17 14:39:33.825445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.825430    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b444b00-eaf2-4666-ad98-1470bed29860" containerName="revoke-keys"
Jul 17 14:39:33.825445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.825437    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10eb41f-52bd-4895-a7de-f160894e10bb" containerName="revoke-keys"
Jul 17 14:39:33.829877 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.829862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"
Jul 17 14:39:33.837941 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.837918    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"]
Jul 17 14:39:33.869946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.869923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314ed04-7eb8-46ad-a6b9-f4ba06c17a63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq\" (UID: \"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"
Jul 17 14:39:33.970965 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.970932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314ed04-7eb8-46ad-a6b9-f4ba06c17a63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq\" (UID: \"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"
Jul 17 14:39:33.971489 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:33.971472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314ed04-7eb8-46ad-a6b9-f4ba06c17a63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq\" (UID: \"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"
Jul 17 14:39:34.141070 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:34.141005    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"
Jul 17 14:39:34.273179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:34.273156    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"]
Jul 17 14:39:34.274763 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:39:34.274733    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc314ed04_7eb8_46ad_a6b9_f4ba06c17a63.slice/crio-7d7c0c4c0a880b29301f76848f0079ef71fdf03ec4090958e0dafb221f62236e WatchSource:0}: Error finding container 7d7c0c4c0a880b29301f76848f0079ef71fdf03ec4090958e0dafb221f62236e: Status 404 returned error can't find the container with id 7d7c0c4c0a880b29301f76848f0079ef71fdf03ec4090958e0dafb221f62236e
Jul 17 14:39:34.817396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:34.817308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" event={"ID":"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63","Type":"ContainerStarted","Data":"46e89d009674b4af277537afc43c7e3520356e02b388dab54e78d6e89d292651"}
Jul 17 14:39:34.817554 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:34.817411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" event={"ID":"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63","Type":"ContainerStarted","Data":"7d7c0c4c0a880b29301f76848f0079ef71fdf03ec4090958e0dafb221f62236e"}
Jul 17 14:39:34.844085 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:34.844042    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" podStartSLOduration=1.844026607 podStartE2EDuration="1.844026607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:39:34.842415353 +0000 UTC m=+3253.229762793" watchObservedRunningTime="2026-07-17 14:39:34.844026607 +0000 UTC m=+3253.231374076"
Jul 17 14:39:35.823002 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:35.822968    2575 generic.go:356] "Generic (PLEG): container finished" podID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerID="46e89d009674b4af277537afc43c7e3520356e02b388dab54e78d6e89d292651" exitCode=6
Jul 17 14:39:35.823191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:35.823059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" event={"ID":"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63","Type":"ContainerDied","Data":"46e89d009674b4af277537afc43c7e3520356e02b388dab54e78d6e89d292651"}
Jul 17 14:39:35.823191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:35.823177    2575 scope.go:117] "RemoveContainer" containerID="46e89d009674b4af277537afc43c7e3520356e02b388dab54e78d6e89d292651"
Jul 17 14:39:36.830136 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:36.830102    2575 generic.go:356] "Generic (PLEG): container finished" podID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerID="f7eed464f9c9b17661a2704545c7191414a1db4e787d4da3f5a14748deae2dea" exitCode=6
Jul 17 14:39:36.830517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:36.830180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" event={"ID":"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63","Type":"ContainerDied","Data":"f7eed464f9c9b17661a2704545c7191414a1db4e787d4da3f5a14748deae2dea"}
Jul 17 14:39:36.830517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:36.830232    2575 scope.go:117] "RemoveContainer" containerID="46e89d009674b4af277537afc43c7e3520356e02b388dab54e78d6e89d292651"
Jul 17 14:39:36.830517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:36.830447    2575 scope.go:117] "RemoveContainer" containerID="f7eed464f9c9b17661a2704545c7191414a1db4e787d4da3f5a14748deae2dea"
Jul 17 14:39:36.830664 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:36.830647    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq_odh-ai-gateway-infra(c314ed04-7eb8-46ad-a6b9-f4ba06c17a63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63"
Jul 17 14:39:37.837580 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:37.837554    2575 scope.go:117] "RemoveContainer" containerID="f7eed464f9c9b17661a2704545c7191414a1db4e787d4da3f5a14748deae2dea"
Jul 17 14:39:37.837932 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:37.837738    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq_odh-ai-gateway-infra(c314ed04-7eb8-46ad-a6b9-f4ba06c17a63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63"
Jul 17 14:39:39.769739 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:39.769706    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"]
Jul 17 14:39:39.774589 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:39.774574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"
Jul 17 14:39:39.781213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:39.781190    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"]
Jul 17 14:39:39.814844 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:39.814816    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18480d46-038a-4731-86bb-d97b6184ae27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7\" (UID: \"18480d46-038a-4731-86bb-d97b6184ae27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"
Jul 17 14:39:39.915837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:39.915809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18480d46-038a-4731-86bb-d97b6184ae27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7\" (UID: \"18480d46-038a-4731-86bb-d97b6184ae27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"
Jul 17 14:39:39.916496 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:39.916479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18480d46-038a-4731-86bb-d97b6184ae27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7\" (UID: \"18480d46-038a-4731-86bb-d97b6184ae27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"
Jul 17 14:39:40.068909 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.068834    2575 scope.go:117] "RemoveContainer" containerID="1d79f7fcef6ee4db9ae8226ecfd0cbc6368c4e0b244d2463b44d7fb7625ae4c0"
Jul 17 14:39:40.085714 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.085692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"
Jul 17 14:39:40.218538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.218508    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"]
Jul 17 14:39:40.220966 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:39:40.220941    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18480d46_038a_4731_86bb_d97b6184ae27.slice/crio-dae43ae493ac051e1c671f7c46ac6f0fd42b56effa896b20c1d306c84aa483fc WatchSource:0}: Error finding container dae43ae493ac051e1c671f7c46ac6f0fd42b56effa896b20c1d306c84aa483fc: Status 404 returned error can't find the container with id dae43ae493ac051e1c671f7c46ac6f0fd42b56effa896b20c1d306c84aa483fc
Jul 17 14:39:40.852007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.851967    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" event={"ID":"18480d46-038a-4731-86bb-d97b6184ae27","Type":"ContainerStarted","Data":"c35993d76506d3661eb039c120a43e591f814948bc08400223bcc4a88d02dbf3"}
Jul 17 14:39:40.852469 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.852062    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" event={"ID":"18480d46-038a-4731-86bb-d97b6184ae27","Type":"ContainerStarted","Data":"dae43ae493ac051e1c671f7c46ac6f0fd42b56effa896b20c1d306c84aa483fc"}
Jul 17 14:39:40.853670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.853649    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c90aa8b-5bab-4228-b259-a4feee902107" containerID="259e36f97ac34e22ce2a78017be10cad098dfb7aa7fb62b3ee560afe484751ca" exitCode=6
Jul 17 14:39:40.853734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.853717    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" event={"ID":"5c90aa8b-5bab-4228-b259-a4feee902107","Type":"ContainerDied","Data":"259e36f97ac34e22ce2a78017be10cad098dfb7aa7fb62b3ee560afe484751ca"}
Jul 17 14:39:40.853783 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.853770    2575 scope.go:117] "RemoveContainer" containerID="1d79f7fcef6ee4db9ae8226ecfd0cbc6368c4e0b244d2463b44d7fb7625ae4c0"
Jul 17 14:39:40.854058 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.854040    2575 scope.go:117] "RemoveContainer" containerID="259e36f97ac34e22ce2a78017be10cad098dfb7aa7fb62b3ee560afe484751ca"
Jul 17 14:39:40.854248 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:40.854232    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-nrmrj_odh-ai-gateway-infra(5c90aa8b-5bab-4228-b259-a4feee902107)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" podUID="5c90aa8b-5bab-4228-b259-a4feee902107"
Jul 17 14:39:40.870585 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:40.870538    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" podStartSLOduration=1.870520973 podStartE2EDuration="1.870520973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:39:40.870179386 +0000 UTC m=+3259.257526838" watchObservedRunningTime="2026-07-17 14:39:40.870520973 +0000 UTC m=+3259.257868413"
Jul 17 14:39:41.861537 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:41.861505    2575 generic.go:356] "Generic (PLEG): container finished" podID="18480d46-038a-4731-86bb-d97b6184ae27" containerID="c35993d76506d3661eb039c120a43e591f814948bc08400223bcc4a88d02dbf3" exitCode=6
Jul 17 14:39:41.861894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:41.861547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" event={"ID":"18480d46-038a-4731-86bb-d97b6184ae27","Type":"ContainerDied","Data":"c35993d76506d3661eb039c120a43e591f814948bc08400223bcc4a88d02dbf3"}
Jul 17 14:39:41.861894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:41.861732    2575 scope.go:117] "RemoveContainer" containerID="c35993d76506d3661eb039c120a43e591f814948bc08400223bcc4a88d02dbf3"
Jul 17 14:39:41.900561 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:41.900528    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"]
Jul 17 14:39:42.033061 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.033039    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"
Jul 17 14:39:42.131462 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.131393    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c90aa8b-5bab-4228-b259-a4feee902107-maas-api-service-ca\") pod \"5c90aa8b-5bab-4228-b259-a4feee902107\" (UID: \"5c90aa8b-5bab-4228-b259-a4feee902107\") "
Jul 17 14:39:42.131779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.131761    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c90aa8b-5bab-4228-b259-a4feee902107-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c90aa8b-5bab-4228-b259-a4feee902107" (UID: "5c90aa8b-5bab-4228-b259-a4feee902107"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:39:42.232925 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.232894    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c90aa8b-5bab-4228-b259-a4feee902107-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:39:42.867940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.867873    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"
Jul 17 14:39:42.867940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.867923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj" event={"ID":"5c90aa8b-5bab-4228-b259-a4feee902107","Type":"ContainerDied","Data":"5b9ae2115c91bfee18febcff8f0c3c3fad318b4a2c81edb2df22b5fd9a4e9031"}
Jul 17 14:39:42.868392 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.867976    2575 scope.go:117] "RemoveContainer" containerID="259e36f97ac34e22ce2a78017be10cad098dfb7aa7fb62b3ee560afe484751ca"
Jul 17 14:39:42.869785 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.869758    2575 generic.go:356] "Generic (PLEG): container finished" podID="18480d46-038a-4731-86bb-d97b6184ae27" containerID="05f755e1727f8e419585fd4608681a01fdf6000636e5bdb5f23b0725ebe9f02a" exitCode=6
Jul 17 14:39:42.869903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.869796    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" event={"ID":"18480d46-038a-4731-86bb-d97b6184ae27","Type":"ContainerDied","Data":"05f755e1727f8e419585fd4608681a01fdf6000636e5bdb5f23b0725ebe9f02a"}
Jul 17 14:39:42.869994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.869981    2575 scope.go:117] "RemoveContainer" containerID="05f755e1727f8e419585fd4608681a01fdf6000636e5bdb5f23b0725ebe9f02a"
Jul 17 14:39:42.870163 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:42.870147    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7_odh-ai-gateway-infra(18480d46-038a-4731-86bb-d97b6184ae27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" podUID="18480d46-038a-4731-86bb-d97b6184ae27"
Jul 17 14:39:42.876778 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.876760    2575 scope.go:117] "RemoveContainer" containerID="c35993d76506d3661eb039c120a43e591f814948bc08400223bcc4a88d02dbf3"
Jul 17 14:39:42.902658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.902631    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"]
Jul 17 14:39:42.905130 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:42.905110    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-nrmrj"]
Jul 17 14:39:43.876623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:43.876599    2575 scope.go:117] "RemoveContainer" containerID="05f755e1727f8e419585fd4608681a01fdf6000636e5bdb5f23b0725ebe9f02a"
Jul 17 14:39:43.877062 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:43.876828    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7_odh-ai-gateway-infra(18480d46-038a-4731-86bb-d97b6184ae27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" podUID="18480d46-038a-4731-86bb-d97b6184ae27"
Jul 17 14:39:44.074649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:44.074622    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" path="/var/lib/kubelet/pods/5c90aa8b-5bab-4228-b259-a4feee902107/volumes"
Jul 17 14:39:52.969732 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.969695    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"]
Jul 17 14:39:52.970110 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.970089    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:39:52.970110 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.970100    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:39:52.970110 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.970110    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:39:52.970211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.970116    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:39:52.970211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.970194    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:39:52.970211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.970205    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:39:52.970211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.970211    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:39:52.974587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.974567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"
Jul 17 14:39:52.980153 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:52.980124    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"]
Jul 17 14:39:53.013295 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.013251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2102b2b-3046-41f3-aa6b-0564f5bdd909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7\" (UID: \"b2102b2b-3046-41f3-aa6b-0564f5bdd909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"
Jul 17 14:39:53.069255 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.069227    2575 scope.go:117] "RemoveContainer" containerID="f7eed464f9c9b17661a2704545c7191414a1db4e787d4da3f5a14748deae2dea"
Jul 17 14:39:53.113820 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.113794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2102b2b-3046-41f3-aa6b-0564f5bdd909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7\" (UID: \"b2102b2b-3046-41f3-aa6b-0564f5bdd909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"
Jul 17 14:39:53.114399 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.114382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2102b2b-3046-41f3-aa6b-0564f5bdd909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7\" (UID: \"b2102b2b-3046-41f3-aa6b-0564f5bdd909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"
Jul 17 14:39:53.286119 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.286098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"
Jul 17 14:39:53.424102 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.424072    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"]
Jul 17 14:39:53.425211 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:39:53.425184    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2102b2b_3046_41f3_aa6b_0564f5bdd909.slice/crio-dcddaba2ae523dc4bea27b257db1322e0d2a253cc64ca3b22250cb19d2b7d26c WatchSource:0}: Error finding container dcddaba2ae523dc4bea27b257db1322e0d2a253cc64ca3b22250cb19d2b7d26c: Status 404 returned error can't find the container with id dcddaba2ae523dc4bea27b257db1322e0d2a253cc64ca3b22250cb19d2b7d26c
Jul 17 14:39:53.929917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.929826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" event={"ID":"b2102b2b-3046-41f3-aa6b-0564f5bdd909","Type":"ContainerStarted","Data":"1b3fcf33240da5ac7c3f2c7eb14c3e489d6ab7b7367ceb9ec1dc410797d11fbf"}
Jul 17 14:39:53.929917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.929880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" event={"ID":"b2102b2b-3046-41f3-aa6b-0564f5bdd909","Type":"ContainerStarted","Data":"dcddaba2ae523dc4bea27b257db1322e0d2a253cc64ca3b22250cb19d2b7d26c"}
Jul 17 14:39:53.931600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.931574    2575 generic.go:356] "Generic (PLEG): container finished" podID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerID="cb5e3e27fa521203345cf81c180e43e634a4b373bc03cfdf517ac77835d0fbd3" exitCode=6
Jul 17 14:39:53.931702 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.931617    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" event={"ID":"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63","Type":"ContainerDied","Data":"cb5e3e27fa521203345cf81c180e43e634a4b373bc03cfdf517ac77835d0fbd3"}
Jul 17 14:39:53.931702 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.931659    2575 scope.go:117] "RemoveContainer" containerID="f7eed464f9c9b17661a2704545c7191414a1db4e787d4da3f5a14748deae2dea"
Jul 17 14:39:53.931911 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.931894    2575 scope.go:117] "RemoveContainer" containerID="cb5e3e27fa521203345cf81c180e43e634a4b373bc03cfdf517ac77835d0fbd3"
Jul 17 14:39:53.932135 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:53.932117    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq_odh-ai-gateway-infra(c314ed04-7eb8-46ad-a6b9-f4ba06c17a63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63"
Jul 17 14:39:53.947663 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:53.947621    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" podStartSLOduration=1.947607012 podStartE2EDuration="1.947607012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:39:53.945531979 +0000 UTC m=+3272.332879420" watchObservedRunningTime="2026-07-17 14:39:53.947607012 +0000 UTC m=+3272.334954452"
Jul 17 14:39:54.939539 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:54.939503    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerID="1b3fcf33240da5ac7c3f2c7eb14c3e489d6ab7b7367ceb9ec1dc410797d11fbf" exitCode=6
Jul 17 14:39:54.939914 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:54.939592    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" event={"ID":"b2102b2b-3046-41f3-aa6b-0564f5bdd909","Type":"ContainerDied","Data":"1b3fcf33240da5ac7c3f2c7eb14c3e489d6ab7b7367ceb9ec1dc410797d11fbf"}
Jul 17 14:39:54.939914 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:54.939759    2575 scope.go:117] "RemoveContainer" containerID="1b3fcf33240da5ac7c3f2c7eb14c3e489d6ab7b7367ceb9ec1dc410797d11fbf"
Jul 17 14:39:54.978662 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:54.978633    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"]
Jul 17 14:39:55.114042 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.114020    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"
Jul 17 14:39:55.129000 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.128980    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314ed04-7eb8-46ad-a6b9-f4ba06c17a63-maas-api-service-ca\") pod \"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63\" (UID: \"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63\") "
Jul 17 14:39:55.129330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.129305    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c314ed04-7eb8-46ad-a6b9-f4ba06c17a63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" (UID: "c314ed04-7eb8-46ad-a6b9-f4ba06c17a63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:39:55.230068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.230048    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314ed04-7eb8-46ad-a6b9-f4ba06c17a63-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:39:55.945734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.945664    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"
Jul 17 14:39:55.945734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.945686    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq" event={"ID":"c314ed04-7eb8-46ad-a6b9-f4ba06c17a63","Type":"ContainerDied","Data":"7d7c0c4c0a880b29301f76848f0079ef71fdf03ec4090958e0dafb221f62236e"}
Jul 17 14:39:55.946225 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.945746    2575 scope.go:117] "RemoveContainer" containerID="cb5e3e27fa521203345cf81c180e43e634a4b373bc03cfdf517ac77835d0fbd3"
Jul 17 14:39:55.947567 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.947539    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerID="ab0389367ca7c6f00b7a64f0c9069cd12f4410bde0ea464a056a9437f8a5db7d" exitCode=6
Jul 17 14:39:55.947673 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.947569    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" event={"ID":"b2102b2b-3046-41f3-aa6b-0564f5bdd909","Type":"ContainerDied","Data":"ab0389367ca7c6f00b7a64f0c9069cd12f4410bde0ea464a056a9437f8a5db7d"}
Jul 17 14:39:55.947816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.947800    2575 scope.go:117] "RemoveContainer" containerID="ab0389367ca7c6f00b7a64f0c9069cd12f4410bde0ea464a056a9437f8a5db7d"
Jul 17 14:39:55.948026 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:55.948007    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7_odh-ai-gateway-infra(b2102b2b-3046-41f3-aa6b-0564f5bdd909)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909"
Jul 17 14:39:55.954771 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.954755    2575 scope.go:117] "RemoveContainer" containerID="1b3fcf33240da5ac7c3f2c7eb14c3e489d6ab7b7367ceb9ec1dc410797d11fbf"
Jul 17 14:39:55.979475 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.979446    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"]
Jul 17 14:39:55.984022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:55.984001    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltfgq"]
Jul 17 14:39:56.069164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:56.069145    2575 scope.go:117] "RemoveContainer" containerID="05f755e1727f8e419585fd4608681a01fdf6000636e5bdb5f23b0725ebe9f02a"
Jul 17 14:39:56.074873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:56.074853    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" path="/var/lib/kubelet/pods/c314ed04-7eb8-46ad-a6b9-f4ba06c17a63/volumes"
Jul 17 14:39:56.954020 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:56.953950    2575 generic.go:356] "Generic (PLEG): container finished" podID="18480d46-038a-4731-86bb-d97b6184ae27" containerID="0d3ecf0c08351838cf84e3bfce7a28b801f9e4d66e562fb37a1bf284a536aa54" exitCode=6
Jul 17 14:39:56.954456 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:56.954017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" event={"ID":"18480d46-038a-4731-86bb-d97b6184ae27","Type":"ContainerDied","Data":"0d3ecf0c08351838cf84e3bfce7a28b801f9e4d66e562fb37a1bf284a536aa54"}
Jul 17 14:39:56.954456 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:56.954072    2575 scope.go:117] "RemoveContainer" containerID="05f755e1727f8e419585fd4608681a01fdf6000636e5bdb5f23b0725ebe9f02a"
Jul 17 14:39:56.954456 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:56.954396    2575 scope.go:117] "RemoveContainer" containerID="0d3ecf0c08351838cf84e3bfce7a28b801f9e4d66e562fb37a1bf284a536aa54"
Jul 17 14:39:56.954646 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:56.954625    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7_odh-ai-gateway-infra(18480d46-038a-4731-86bb-d97b6184ae27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" podUID="18480d46-038a-4731-86bb-d97b6184ae27"
Jul 17 14:39:56.956778 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:56.956759    2575 scope.go:117] "RemoveContainer" containerID="ab0389367ca7c6f00b7a64f0c9069cd12f4410bde0ea464a056a9437f8a5db7d"
Jul 17 14:39:56.956981 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:39:56.956961    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7_odh-ai-gateway-infra(b2102b2b-3046-41f3-aa6b-0564f5bdd909)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909"
Jul 17 14:39:57.981917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:57.981879    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"]
Jul 17 14:39:58.109917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.109892    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"
Jul 17 14:39:58.153135 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.153111    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18480d46-038a-4731-86bb-d97b6184ae27-maas-api-service-ca\") pod \"18480d46-038a-4731-86bb-d97b6184ae27\" (UID: \"18480d46-038a-4731-86bb-d97b6184ae27\") "
Jul 17 14:39:58.153517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.153495    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18480d46-038a-4731-86bb-d97b6184ae27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18480d46-038a-4731-86bb-d97b6184ae27" (UID: "18480d46-038a-4731-86bb-d97b6184ae27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:39:58.254332 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.254239    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18480d46-038a-4731-86bb-d97b6184ae27-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:39:58.969533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.969506    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"
Jul 17 14:39:58.969533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.969517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7" event={"ID":"18480d46-038a-4731-86bb-d97b6184ae27","Type":"ContainerDied","Data":"dae43ae493ac051e1c671f7c46ac6f0fd42b56effa896b20c1d306c84aa483fc"}
Jul 17 14:39:58.969759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.969572    2575 scope.go:117] "RemoveContainer" containerID="0d3ecf0c08351838cf84e3bfce7a28b801f9e4d66e562fb37a1bf284a536aa54"
Jul 17 14:39:58.991829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.991802    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"]
Jul 17 14:39:58.995367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:39:58.995348    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-vl7w7"]
Jul 17 14:40:00.074906 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:00.074876    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18480d46-038a-4731-86bb-d97b6184ae27" path="/var/lib/kubelet/pods/18480d46-038a-4731-86bb-d97b6184ae27/volumes"
Jul 17 14:40:08.069595 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:08.069564    2575 scope.go:117] "RemoveContainer" containerID="ab0389367ca7c6f00b7a64f0c9069cd12f4410bde0ea464a056a9437f8a5db7d"
Jul 17 14:40:09.031023 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:09.030990    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerID="9b94d8594d35499498b4e5c17865703023ccc79c63dc476c1afa476918a21c65" exitCode=6
Jul 17 14:40:09.031190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:09.031044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" event={"ID":"b2102b2b-3046-41f3-aa6b-0564f5bdd909","Type":"ContainerDied","Data":"9b94d8594d35499498b4e5c17865703023ccc79c63dc476c1afa476918a21c65"}
Jul 17 14:40:09.031190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:09.031090    2575 scope.go:117] "RemoveContainer" containerID="ab0389367ca7c6f00b7a64f0c9069cd12f4410bde0ea464a056a9437f8a5db7d"
Jul 17 14:40:09.031359 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:09.031343    2575 scope.go:117] "RemoveContainer" containerID="9b94d8594d35499498b4e5c17865703023ccc79c63dc476c1afa476918a21c65"
Jul 17 14:40:09.031580 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:09.031560    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7_odh-ai-gateway-infra(b2102b2b-3046-41f3-aa6b-0564f5bdd909)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909"
Jul 17 14:40:10.063621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.063586    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"]
Jul 17 14:40:10.111247 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111216    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x6pkf"]
Jul 17 14:40:10.111682 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111663    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111683    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111715    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111724    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111738    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111746    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c90aa8b-5bab-4228-b259-a4feee902107" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111755    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111763    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111771    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:10.111776 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111779    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:10.112191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111794    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.112191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111802    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.112191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111903    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:10.112191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111916    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.112191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111926    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:10.112191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111935    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:10.112191 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.111946    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:10.116519 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.116337    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:10.119806 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.119211    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 14:40:10.119806 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.119442    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 14:40:10.119991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.119969    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zwglj\""
Jul 17 14:40:10.124531 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.124503    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x6pkf"]
Jul 17 14:40:10.188091 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.188068    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"
Jul 17 14:40:10.250368 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.250341    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2102b2b-3046-41f3-aa6b-0564f5bdd909-maas-api-service-ca\") pod \"b2102b2b-3046-41f3-aa6b-0564f5bdd909\" (UID: \"b2102b2b-3046-41f3-aa6b-0564f5bdd909\") "
Jul 17 14:40:10.250541 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.250505    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5bx\" (UniqueName: \"kubernetes.io/projected/f5e3b977-79c5-4cad-ba16-6faac07ffe01-kube-api-access-7x5bx\") pod \"kuadrant-operator-catalog-x6pkf\" (UID: \"f5e3b977-79c5-4cad-ba16-6faac07ffe01\") " pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:10.250718 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.250695    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2102b2b-3046-41f3-aa6b-0564f5bdd909-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2102b2b-3046-41f3-aa6b-0564f5bdd909" (UID: "b2102b2b-3046-41f3-aa6b-0564f5bdd909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:40:10.351709 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.351637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5bx\" (UniqueName: \"kubernetes.io/projected/f5e3b977-79c5-4cad-ba16-6faac07ffe01-kube-api-access-7x5bx\") pod \"kuadrant-operator-catalog-x6pkf\" (UID: \"f5e3b977-79c5-4cad-ba16-6faac07ffe01\") " pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:10.351806 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.351730    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2102b2b-3046-41f3-aa6b-0564f5bdd909-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:40:10.360148 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.360127    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5bx\" (UniqueName: \"kubernetes.io/projected/f5e3b977-79c5-4cad-ba16-6faac07ffe01-kube-api-access-7x5bx\") pod \"kuadrant-operator-catalog-x6pkf\" (UID: \"f5e3b977-79c5-4cad-ba16-6faac07ffe01\") " pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:10.431451 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.431430    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:10.563671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:10.563647    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x6pkf"]
Jul 17 14:40:10.565986 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:40:10.565961    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e3b977_79c5_4cad_ba16_6faac07ffe01.slice/crio-c576d439c3a7e712c9f4cf9e3c8e2bf7c9ba826ae3e2c2835190df487bd783aa WatchSource:0}: Error finding container c576d439c3a7e712c9f4cf9e3c8e2bf7c9ba826ae3e2c2835190df487bd783aa: Status 404 returned error can't find the container with id c576d439c3a7e712c9f4cf9e3c8e2bf7c9ba826ae3e2c2835190df487bd783aa
Jul 17 14:40:11.044850 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:11.044817    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf" event={"ID":"f5e3b977-79c5-4cad-ba16-6faac07ffe01","Type":"ContainerStarted","Data":"c576d439c3a7e712c9f4cf9e3c8e2bf7c9ba826ae3e2c2835190df487bd783aa"}
Jul 17 14:40:11.046124 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:11.046103    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"
Jul 17 14:40:11.046234 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:11.046138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7" event={"ID":"b2102b2b-3046-41f3-aa6b-0564f5bdd909","Type":"ContainerDied","Data":"dcddaba2ae523dc4bea27b257db1322e0d2a253cc64ca3b22250cb19d2b7d26c"}
Jul 17 14:40:11.046234 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:11.046196    2575 scope.go:117] "RemoveContainer" containerID="9b94d8594d35499498b4e5c17865703023ccc79c63dc476c1afa476918a21c65"
Jul 17 14:40:11.072202 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:11.072167    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"]
Jul 17 14:40:11.084736 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:11.084711    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-tlsh7"]
Jul 17 14:40:12.076314 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:12.076289    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" path="/var/lib/kubelet/pods/b2102b2b-3046-41f3-aa6b-0564f5bdd909/volumes"
Jul 17 14:40:13.058456 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:13.058418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf" event={"ID":"f5e3b977-79c5-4cad-ba16-6faac07ffe01","Type":"ContainerStarted","Data":"8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98"}
Jul 17 14:40:13.076150 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:13.076108    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf" podStartSLOduration=1.592971261 podStartE2EDuration="3.076095171s" totalImagesPullingTime="1.48312391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:40:13.075110945 +0000 UTC m=+3291.462458385" watchObservedRunningTime="2026-07-17 14:40:13.076095171 +0000 UTC m=+3291.463442627"
Jul 17 14:40:20.432318 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:20.432287    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:20.432719 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:20.432358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:20.454316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:20.454293    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:21.121266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:21.121239    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:21.165624 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:21.165592    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x6pkf"]
Jul 17 14:40:22.257747 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.257716    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"]
Jul 17 14:40:22.258209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258127    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:22.258209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258143    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="18480d46-038a-4731-86bb-d97b6184ae27" containerName="revoke-keys"
Jul 17 14:40:22.258209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258157    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:22.258209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258162    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:22.258209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258169    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:22.258209 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258174    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:22.258447 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258252    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c314ed04-7eb8-46ad-a6b9-f4ba06c17a63" containerName="revoke-keys"
Jul 17 14:40:22.258447 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258260    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:22.258447 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.258267    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:22.261798 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.261770    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"
Jul 17 14:40:22.264029 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.264008    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:40:22.269506 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.269483    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"]
Jul 17 14:40:22.275640 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.275620    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:40:22.280652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.280623    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:40:22.287517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.287498    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:40:22.290468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.290453    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:40:22.294479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.294464    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:40:22.300868 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.300853    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:40:22.358753 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.357654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82de0c-557e-4be6-8518-4a13f9824acd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-4xlhk\" (UID: \"7e82de0c-557e-4be6-8518-4a13f9824acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"
Jul 17 14:40:22.458900 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.458872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82de0c-557e-4be6-8518-4a13f9824acd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-4xlhk\" (UID: \"7e82de0c-557e-4be6-8518-4a13f9824acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"
Jul 17 14:40:22.459478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.459459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82de0c-557e-4be6-8518-4a13f9824acd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-4xlhk\" (UID: \"7e82de0c-557e-4be6-8518-4a13f9824acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"
Jul 17 14:40:22.575148 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.575093    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"
Jul 17 14:40:22.733156 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:22.733116    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"]
Jul 17 14:40:22.734832 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:40:22.734802    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e82de0c_557e_4be6_8518_4a13f9824acd.slice/crio-4e77535fefacd8f5a15b1ef9cd762d594e72afb47b87c67a07cbfaf020554d8a WatchSource:0}: Error finding container 4e77535fefacd8f5a15b1ef9cd762d594e72afb47b87c67a07cbfaf020554d8a: Status 404 returned error can't find the container with id 4e77535fefacd8f5a15b1ef9cd762d594e72afb47b87c67a07cbfaf020554d8a
Jul 17 14:40:23.109910 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.109879    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" event={"ID":"7e82de0c-557e-4be6-8518-4a13f9824acd","Type":"ContainerStarted","Data":"a09b0dec208c9dc0d6a407f21ddbcb70eafc842fc349e435efec48000a8a3484"}
Jul 17 14:40:23.110035 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.109930    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" event={"ID":"7e82de0c-557e-4be6-8518-4a13f9824acd","Type":"ContainerStarted","Data":"4e77535fefacd8f5a15b1ef9cd762d594e72afb47b87c67a07cbfaf020554d8a"}
Jul 17 14:40:23.110104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.110080    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf" podUID="f5e3b977-79c5-4cad-ba16-6faac07ffe01" containerName="registry-server" containerID="cri-o://8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98" gracePeriod=2
Jul 17 14:40:23.129071 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.129032    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" podStartSLOduration=1.129018466 podStartE2EDuration="1.129018466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:40:23.127663057 +0000 UTC m=+3301.515010496" watchObservedRunningTime="2026-07-17 14:40:23.129018466 +0000 UTC m=+3301.516365940"
Jul 17 14:40:23.353964 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.353933    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:23.467533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.467508    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x5bx\" (UniqueName: \"kubernetes.io/projected/f5e3b977-79c5-4cad-ba16-6faac07ffe01-kube-api-access-7x5bx\") pod \"f5e3b977-79c5-4cad-ba16-6faac07ffe01\" (UID: \"f5e3b977-79c5-4cad-ba16-6faac07ffe01\") "
Jul 17 14:40:23.469422 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.469397    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e3b977-79c5-4cad-ba16-6faac07ffe01-kube-api-access-7x5bx" (OuterVolumeSpecName: "kube-api-access-7x5bx") pod "f5e3b977-79c5-4cad-ba16-6faac07ffe01" (UID: "f5e3b977-79c5-4cad-ba16-6faac07ffe01"). InnerVolumeSpecName "kube-api-access-7x5bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:40:23.568652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:23.568593    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5bx\" (UniqueName: \"kubernetes.io/projected/f5e3b977-79c5-4cad-ba16-6faac07ffe01-kube-api-access-7x5bx\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:40:24.114981 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.114952    2575 generic.go:356] "Generic (PLEG): container finished" podID="f5e3b977-79c5-4cad-ba16-6faac07ffe01" containerID="8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98" exitCode=0
Jul 17 14:40:24.115120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.114993    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf"
Jul 17 14:40:24.115120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.115038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf" event={"ID":"f5e3b977-79c5-4cad-ba16-6faac07ffe01","Type":"ContainerDied","Data":"8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98"}
Jul 17 14:40:24.115120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.115088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x6pkf" event={"ID":"f5e3b977-79c5-4cad-ba16-6faac07ffe01","Type":"ContainerDied","Data":"c576d439c3a7e712c9f4cf9e3c8e2bf7c9ba826ae3e2c2835190df487bd783aa"}
Jul 17 14:40:24.115120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.115118    2575 scope.go:117] "RemoveContainer" containerID="8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98"
Jul 17 14:40:24.117101 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.117071    2575 generic.go:356] "Generic (PLEG): container finished" podID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerID="a09b0dec208c9dc0d6a407f21ddbcb70eafc842fc349e435efec48000a8a3484" exitCode=6
Jul 17 14:40:24.117231 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.117213    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" event={"ID":"7e82de0c-557e-4be6-8518-4a13f9824acd","Type":"ContainerDied","Data":"a09b0dec208c9dc0d6a407f21ddbcb70eafc842fc349e435efec48000a8a3484"}
Jul 17 14:40:24.117400 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.117385    2575 scope.go:117] "RemoveContainer" containerID="a09b0dec208c9dc0d6a407f21ddbcb70eafc842fc349e435efec48000a8a3484"
Jul 17 14:40:24.124044 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.124024    2575 scope.go:117] "RemoveContainer" containerID="8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98"
Jul 17 14:40:24.124306 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:24.124272    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98\": container with ID starting with 8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98 not found: ID does not exist" containerID="8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98"
Jul 17 14:40:24.124385 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.124313    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98"} err="failed to get container status \"8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98\": rpc error: code = NotFound desc = could not find container \"8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98\": container with ID starting with 8ae063a169d2cc11b2b48d0680ac78713a00f4185d0b9e97b24c65d19ca0bb98 not found: ID does not exist"
Jul 17 14:40:24.148243 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.148223    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x6pkf"]
Jul 17 14:40:24.154059 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:24.154040    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x6pkf"]
Jul 17 14:40:25.124386 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:25.124354    2575 generic.go:356] "Generic (PLEG): container finished" podID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerID="2c71bfbd0b59b7c41b72da6a6fe05750bc4691d3c7578211fc64c85ad8753f76" exitCode=6
Jul 17 14:40:25.124821 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:25.124438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" event={"ID":"7e82de0c-557e-4be6-8518-4a13f9824acd","Type":"ContainerDied","Data":"2c71bfbd0b59b7c41b72da6a6fe05750bc4691d3c7578211fc64c85ad8753f76"}
Jul 17 14:40:25.124821 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:25.124501    2575 scope.go:117] "RemoveContainer" containerID="a09b0dec208c9dc0d6a407f21ddbcb70eafc842fc349e435efec48000a8a3484"
Jul 17 14:40:25.124821 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:25.124673    2575 scope.go:117] "RemoveContainer" containerID="2c71bfbd0b59b7c41b72da6a6fe05750bc4691d3c7578211fc64c85ad8753f76"
Jul 17 14:40:25.124948 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:25.124898    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-4xlhk_odh-ai-gateway-infra(7e82de0c-557e-4be6-8518-4a13f9824acd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd"
Jul 17 14:40:26.074882 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:26.074852    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e3b977-79c5-4cad-ba16-6faac07ffe01" path="/var/lib/kubelet/pods/f5e3b977-79c5-4cad-ba16-6faac07ffe01/volumes"
Jul 17 14:40:26.130662 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:26.130639    2575 scope.go:117] "RemoveContainer" containerID="2c71bfbd0b59b7c41b72da6a6fe05750bc4691d3c7578211fc64c85ad8753f76"
Jul 17 14:40:26.130995 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:26.130792    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-4xlhk_odh-ai-gateway-infra(7e82de0c-557e-4be6-8518-4a13f9824acd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd"
Jul 17 14:40:29.819835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.819807    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"]
Jul 17 14:40:29.820204 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.820191    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e3b977-79c5-4cad-ba16-6faac07ffe01" containerName="registry-server"
Jul 17 14:40:29.820204 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.820205    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e3b977-79c5-4cad-ba16-6faac07ffe01" containerName="registry-server"
Jul 17 14:40:29.820320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.820221    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:29.820320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.820227    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:29.820320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.820316    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2102b2b-3046-41f3-aa6b-0564f5bdd909" containerName="revoke-keys"
Jul 17 14:40:29.820415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.820327    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e3b977-79c5-4cad-ba16-6faac07ffe01" containerName="registry-server"
Jul 17 14:40:29.824705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.824688    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"
Jul 17 14:40:29.834576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.834549    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"]
Jul 17 14:40:29.925786 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:29.925752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ece7f9-d23b-4666-b0a3-47c4c6d25977-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj\" (UID: \"83ece7f9-d23b-4666-b0a3-47c4c6d25977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"
Jul 17 14:40:30.026596 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:30.026563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ece7f9-d23b-4666-b0a3-47c4c6d25977-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj\" (UID: \"83ece7f9-d23b-4666-b0a3-47c4c6d25977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"
Jul 17 14:40:30.027136 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:30.027116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ece7f9-d23b-4666-b0a3-47c4c6d25977-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj\" (UID: \"83ece7f9-d23b-4666-b0a3-47c4c6d25977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"
Jul 17 14:40:30.136097 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:30.136025    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"
Jul 17 14:40:30.264905 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:30.264882    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"]
Jul 17 14:40:30.267023 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:40:30.266994    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ece7f9_d23b_4666_b0a3_47c4c6d25977.slice/crio-7f1ce18007658f7aefe8a4e02cd41c92dd7438667d82ff3b673c8735e1d3952a WatchSource:0}: Error finding container 7f1ce18007658f7aefe8a4e02cd41c92dd7438667d82ff3b673c8735e1d3952a: Status 404 returned error can't find the container with id 7f1ce18007658f7aefe8a4e02cd41c92dd7438667d82ff3b673c8735e1d3952a
Jul 17 14:40:31.154934 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:31.154903    2575 generic.go:356] "Generic (PLEG): container finished" podID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerID="030fe5c71354e48b38cc74b6210d391ff11545bd23f8052f00998a2f94ab8430" exitCode=6
Jul 17 14:40:31.155323 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:31.154988    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" event={"ID":"83ece7f9-d23b-4666-b0a3-47c4c6d25977","Type":"ContainerDied","Data":"030fe5c71354e48b38cc74b6210d391ff11545bd23f8052f00998a2f94ab8430"}
Jul 17 14:40:31.155323 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:31.155037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" event={"ID":"83ece7f9-d23b-4666-b0a3-47c4c6d25977","Type":"ContainerStarted","Data":"7f1ce18007658f7aefe8a4e02cd41c92dd7438667d82ff3b673c8735e1d3952a"}
Jul 17 14:40:31.155323 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:31.155141    2575 scope.go:117] "RemoveContainer" containerID="030fe5c71354e48b38cc74b6210d391ff11545bd23f8052f00998a2f94ab8430"
Jul 17 14:40:32.161476 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:32.161400    2575 generic.go:356] "Generic (PLEG): container finished" podID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerID="0c4330ae164a44185b1a712aea6a4952a1154b6b58353d0175adc11410541bf1" exitCode=6
Jul 17 14:40:32.161875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:32.161492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" event={"ID":"83ece7f9-d23b-4666-b0a3-47c4c6d25977","Type":"ContainerDied","Data":"0c4330ae164a44185b1a712aea6a4952a1154b6b58353d0175adc11410541bf1"}
Jul 17 14:40:32.161875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:32.161559    2575 scope.go:117] "RemoveContainer" containerID="030fe5c71354e48b38cc74b6210d391ff11545bd23f8052f00998a2f94ab8430"
Jul 17 14:40:32.161875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:32.161608    2575 scope.go:117] "RemoveContainer" containerID="0c4330ae164a44185b1a712aea6a4952a1154b6b58353d0175adc11410541bf1"
Jul 17 14:40:32.161875 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:32.161771    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj_odh-ai-gateway-infra(83ece7f9-d23b-4666-b0a3-47c4c6d25977)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977"
Jul 17 14:40:33.167908 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:33.167883    2575 scope.go:117] "RemoveContainer" containerID="0c4330ae164a44185b1a712aea6a4952a1154b6b58353d0175adc11410541bf1"
Jul 17 14:40:33.168299 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:33.168061    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj_odh-ai-gateway-infra(83ece7f9-d23b-4666-b0a3-47c4c6d25977)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977"
Jul 17 14:40:33.874587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:33.874558    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"]
Jul 17 14:40:33.879264 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:33.879248    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"
Jul 17 14:40:33.884849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:33.884823    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"]
Jul 17 14:40:33.965187 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:33.965147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83da60ca-94d1-491a-b2cb-b4db1e5559f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b\" (UID: \"83da60ca-94d1-491a-b2cb-b4db1e5559f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"
Jul 17 14:40:34.066421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:34.066396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83da60ca-94d1-491a-b2cb-b4db1e5559f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b\" (UID: \"83da60ca-94d1-491a-b2cb-b4db1e5559f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"
Jul 17 14:40:34.067022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:34.067000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83da60ca-94d1-491a-b2cb-b4db1e5559f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b\" (UID: \"83da60ca-94d1-491a-b2cb-b4db1e5559f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"
Jul 17 14:40:34.189991 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:34.189926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"
Jul 17 14:40:34.316179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:34.316150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"]
Jul 17 14:40:34.319830 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:40:34.319799    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83da60ca_94d1_491a_b2cb_b4db1e5559f1.slice/crio-739d7191c5e7d079d44751ec9f9a589c47d40efc241888b8b3fd321e670cd7d8 WatchSource:0}: Error finding container 739d7191c5e7d079d44751ec9f9a589c47d40efc241888b8b3fd321e670cd7d8: Status 404 returned error can't find the container with id 739d7191c5e7d079d44751ec9f9a589c47d40efc241888b8b3fd321e670cd7d8
Jul 17 14:40:35.178625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:35.178599    2575 generic.go:356] "Generic (PLEG): container finished" podID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerID="7e557d5ac5abc15db1a5924116077056b319387b83537a484a1d2eed501bb4ca" exitCode=6
Jul 17 14:40:35.178740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:35.178658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" event={"ID":"83da60ca-94d1-491a-b2cb-b4db1e5559f1","Type":"ContainerDied","Data":"7e557d5ac5abc15db1a5924116077056b319387b83537a484a1d2eed501bb4ca"}
Jul 17 14:40:35.178740 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:35.178718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" event={"ID":"83da60ca-94d1-491a-b2cb-b4db1e5559f1","Type":"ContainerStarted","Data":"739d7191c5e7d079d44751ec9f9a589c47d40efc241888b8b3fd321e670cd7d8"}
Jul 17 14:40:35.178933 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:35.178920    2575 scope.go:117] "RemoveContainer" containerID="7e557d5ac5abc15db1a5924116077056b319387b83537a484a1d2eed501bb4ca"
Jul 17 14:40:36.185043 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:36.184973    2575 generic.go:356] "Generic (PLEG): container finished" podID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerID="963c2cf3bfa113f7f0f301065a1aff388d4ea7e31ef6fb32c9d256738c85ba8c" exitCode=6
Jul 17 14:40:36.185043 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:36.185010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" event={"ID":"83da60ca-94d1-491a-b2cb-b4db1e5559f1","Type":"ContainerDied","Data":"963c2cf3bfa113f7f0f301065a1aff388d4ea7e31ef6fb32c9d256738c85ba8c"}
Jul 17 14:40:36.185460 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:36.185070    2575 scope.go:117] "RemoveContainer" containerID="7e557d5ac5abc15db1a5924116077056b319387b83537a484a1d2eed501bb4ca"
Jul 17 14:40:36.185460 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:36.185233    2575 scope.go:117] "RemoveContainer" containerID="963c2cf3bfa113f7f0f301065a1aff388d4ea7e31ef6fb32c9d256738c85ba8c"
Jul 17 14:40:36.185460 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:36.185444    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b_odh-ai-gateway-infra(83da60ca-94d1-491a-b2cb-b4db1e5559f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1"
Jul 17 14:40:37.069105 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:37.069079    2575 scope.go:117] "RemoveContainer" containerID="2c71bfbd0b59b7c41b72da6a6fe05750bc4691d3c7578211fc64c85ad8753f76"
Jul 17 14:40:37.191871 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:37.191846    2575 scope.go:117] "RemoveContainer" containerID="963c2cf3bfa113f7f0f301065a1aff388d4ea7e31ef6fb32c9d256738c85ba8c"
Jul 17 14:40:37.192188 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:37.192013    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b_odh-ai-gateway-infra(83da60ca-94d1-491a-b2cb-b4db1e5559f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1"
Jul 17 14:40:38.198710 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:38.198684    2575 generic.go:356] "Generic (PLEG): container finished" podID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerID="b3f935447fb2fdfa20cf12b2793e9424ee0d95780bdb4db02326d2ee78de1d79" exitCode=6
Jul 17 14:40:38.199128 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:38.198770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" event={"ID":"7e82de0c-557e-4be6-8518-4a13f9824acd","Type":"ContainerDied","Data":"b3f935447fb2fdfa20cf12b2793e9424ee0d95780bdb4db02326d2ee78de1d79"}
Jul 17 14:40:38.199128 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:38.198832    2575 scope.go:117] "RemoveContainer" containerID="2c71bfbd0b59b7c41b72da6a6fe05750bc4691d3c7578211fc64c85ad8753f76"
Jul 17 14:40:38.199128 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:38.199104    2575 scope.go:117] "RemoveContainer" containerID="b3f935447fb2fdfa20cf12b2793e9424ee0d95780bdb4db02326d2ee78de1d79"
Jul 17 14:40:38.199383 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:38.199348    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-4xlhk_odh-ai-gateway-infra(7e82de0c-557e-4be6-8518-4a13f9824acd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd"
Jul 17 14:40:39.265817 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:39.265786    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"]
Jul 17 14:40:39.400296 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:39.400253    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"
Jul 17 14:40:39.510260 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:39.510232    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82de0c-557e-4be6-8518-4a13f9824acd-maas-api-service-ca\") pod \"7e82de0c-557e-4be6-8518-4a13f9824acd\" (UID: \"7e82de0c-557e-4be6-8518-4a13f9824acd\") "
Jul 17 14:40:39.510577 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:39.510555    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e82de0c-557e-4be6-8518-4a13f9824acd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e82de0c-557e-4be6-8518-4a13f9824acd" (UID: "7e82de0c-557e-4be6-8518-4a13f9824acd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:40:39.611489 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:39.611432    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82de0c-557e-4be6-8518-4a13f9824acd-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:40:40.211266 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:40.211240    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"
Jul 17 14:40:40.211432 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:40.211240    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk" event={"ID":"7e82de0c-557e-4be6-8518-4a13f9824acd","Type":"ContainerDied","Data":"4e77535fefacd8f5a15b1ef9cd762d594e72afb47b87c67a07cbfaf020554d8a"}
Jul 17 14:40:40.211432 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:40.211373    2575 scope.go:117] "RemoveContainer" containerID="b3f935447fb2fdfa20cf12b2793e9424ee0d95780bdb4db02326d2ee78de1d79"
Jul 17 14:40:40.233841 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:40.233818    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"]
Jul 17 14:40:40.244302 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:40.240798    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-4xlhk"]
Jul 17 14:40:42.074916 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:42.074884    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" path="/var/lib/kubelet/pods/7e82de0c-557e-4be6-8518-4a13f9824acd/volumes"
Jul 17 14:40:43.015198 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.015167    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"]
Jul 17 14:40:43.015587 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.015573    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:40:43.015647 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.015589    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:40:43.015647 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.015616    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:40:43.015647 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.015621    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:40:43.015755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.015682    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:40:43.015755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.015695    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:40:43.021399 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.021379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"
Jul 17 14:40:43.030533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.030511    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"]
Jul 17 14:40:43.136692 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.136663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da71a8b-e839-499f-b1b4-b7075d5c1958-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz\" (UID: \"1da71a8b-e839-499f-b1b4-b7075d5c1958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"
Jul 17 14:40:43.237459 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.237435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da71a8b-e839-499f-b1b4-b7075d5c1958-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz\" (UID: \"1da71a8b-e839-499f-b1b4-b7075d5c1958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"
Jul 17 14:40:43.238026 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.238001    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da71a8b-e839-499f-b1b4-b7075d5c1958-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz\" (UID: \"1da71a8b-e839-499f-b1b4-b7075d5c1958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"
Jul 17 14:40:43.332463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.332412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"
Jul 17 14:40:43.459268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:43.459242    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"]
Jul 17 14:40:43.461976 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:40:43.461945    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da71a8b_e839_499f_b1b4_b7075d5c1958.slice/crio-89d6efb58d87995d0d2eb31e0daeb8967515f8ec1d1251310a5061b4fb1cab22 WatchSource:0}: Error finding container 89d6efb58d87995d0d2eb31e0daeb8967515f8ec1d1251310a5061b4fb1cab22: Status 404 returned error can't find the container with id 89d6efb58d87995d0d2eb31e0daeb8967515f8ec1d1251310a5061b4fb1cab22
Jul 17 14:40:44.232721 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:44.232659    2575 generic.go:356] "Generic (PLEG): container finished" podID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerID="0f906af13e1e686375fd43da48db1837fb74fd82924912d7bd0586de4146433e" exitCode=6
Jul 17 14:40:44.232721 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:44.232709    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" event={"ID":"1da71a8b-e839-499f-b1b4-b7075d5c1958","Type":"ContainerDied","Data":"0f906af13e1e686375fd43da48db1837fb74fd82924912d7bd0586de4146433e"}
Jul 17 14:40:44.233057 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:44.232743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" event={"ID":"1da71a8b-e839-499f-b1b4-b7075d5c1958","Type":"ContainerStarted","Data":"89d6efb58d87995d0d2eb31e0daeb8967515f8ec1d1251310a5061b4fb1cab22"}
Jul 17 14:40:44.233057 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:44.232867    2575 scope.go:117] "RemoveContainer" containerID="0f906af13e1e686375fd43da48db1837fb74fd82924912d7bd0586de4146433e"
Jul 17 14:40:45.239904 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:45.239877    2575 generic.go:356] "Generic (PLEG): container finished" podID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerID="19bdb7653545dc1de626f866873941ab141c7d71fd56fc8f25101101a83d266b" exitCode=6
Jul 17 14:40:45.240259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:45.239939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" event={"ID":"1da71a8b-e839-499f-b1b4-b7075d5c1958","Type":"ContainerDied","Data":"19bdb7653545dc1de626f866873941ab141c7d71fd56fc8f25101101a83d266b"}
Jul 17 14:40:45.240259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:45.239997    2575 scope.go:117] "RemoveContainer" containerID="0f906af13e1e686375fd43da48db1837fb74fd82924912d7bd0586de4146433e"
Jul 17 14:40:45.240259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:45.240228    2575 scope.go:117] "RemoveContainer" containerID="19bdb7653545dc1de626f866873941ab141c7d71fd56fc8f25101101a83d266b"
Jul 17 14:40:45.240450 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:45.240432    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz_odh-ai-gateway-infra(1da71a8b-e839-499f-b1b4-b7075d5c1958)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958"
Jul 17 14:40:46.246743 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:46.246716    2575 scope.go:117] "RemoveContainer" containerID="19bdb7653545dc1de626f866873941ab141c7d71fd56fc8f25101101a83d266b"
Jul 17 14:40:46.247099 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:46.246887    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz_odh-ai-gateway-infra(1da71a8b-e839-499f-b1b4-b7075d5c1958)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958"
Jul 17 14:40:48.068975 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:48.068946    2575 scope.go:117] "RemoveContainer" containerID="0c4330ae164a44185b1a712aea6a4952a1154b6b58353d0175adc11410541bf1"
Jul 17 14:40:49.069631 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:49.069604    2575 scope.go:117] "RemoveContainer" containerID="963c2cf3bfa113f7f0f301065a1aff388d4ea7e31ef6fb32c9d256738c85ba8c"
Jul 17 14:40:49.266445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:49.266419    2575 generic.go:356] "Generic (PLEG): container finished" podID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerID="b82dbb6297469b0acd1c69aafea9e8cbfda52d650bc2e0b51801474804894cb0" exitCode=6
Jul 17 14:40:49.266570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:49.266487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" event={"ID":"83ece7f9-d23b-4666-b0a3-47c4c6d25977","Type":"ContainerDied","Data":"b82dbb6297469b0acd1c69aafea9e8cbfda52d650bc2e0b51801474804894cb0"}
Jul 17 14:40:49.266570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:49.266545    2575 scope.go:117] "RemoveContainer" containerID="0c4330ae164a44185b1a712aea6a4952a1154b6b58353d0175adc11410541bf1"
Jul 17 14:40:49.266877 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:49.266863    2575 scope.go:117] "RemoveContainer" containerID="b82dbb6297469b0acd1c69aafea9e8cbfda52d650bc2e0b51801474804894cb0"
Jul 17 14:40:49.267085 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:49.267068    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj_odh-ai-gateway-infra(83ece7f9-d23b-4666-b0a3-47c4c6d25977)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977"
Jul 17 14:40:50.274620 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.274589    2575 generic.go:356] "Generic (PLEG): container finished" podID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerID="9f2f481bfd998efeca3a85c91737d64d64c144768a1875ce1f0893bf71ade234" exitCode=6
Jul 17 14:40:50.275004 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.274664    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" event={"ID":"83da60ca-94d1-491a-b2cb-b4db1e5559f1","Type":"ContainerDied","Data":"9f2f481bfd998efeca3a85c91737d64d64c144768a1875ce1f0893bf71ade234"}
Jul 17 14:40:50.275004 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.274723    2575 scope.go:117] "RemoveContainer" containerID="963c2cf3bfa113f7f0f301065a1aff388d4ea7e31ef6fb32c9d256738c85ba8c"
Jul 17 14:40:50.275004 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.274998    2575 scope.go:117] "RemoveContainer" containerID="9f2f481bfd998efeca3a85c91737d64d64c144768a1875ce1f0893bf71ade234"
Jul 17 14:40:50.275206 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:40:50.275185    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b_odh-ai-gateway-infra(83da60ca-94d1-491a-b2cb-b4db1e5559f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1"
Jul 17 14:40:50.291452 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.291430    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"]
Jul 17 14:40:50.423982 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.423957    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"
Jul 17 14:40:50.495322 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.495295    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ece7f9-d23b-4666-b0a3-47c4c6d25977-maas-api-service-ca\") pod \"83ece7f9-d23b-4666-b0a3-47c4c6d25977\" (UID: \"83ece7f9-d23b-4666-b0a3-47c4c6d25977\") "
Jul 17 14:40:50.495625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.495601    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ece7f9-d23b-4666-b0a3-47c4c6d25977-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83ece7f9-d23b-4666-b0a3-47c4c6d25977" (UID: "83ece7f9-d23b-4666-b0a3-47c4c6d25977"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:40:50.596190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:50.596128    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ece7f9-d23b-4666-b0a3-47c4c6d25977-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:40:51.281214 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.281184    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"
Jul 17 14:40:51.281214 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.281194    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj" event={"ID":"83ece7f9-d23b-4666-b0a3-47c4c6d25977","Type":"ContainerDied","Data":"7f1ce18007658f7aefe8a4e02cd41c92dd7438667d82ff3b673c8735e1d3952a"}
Jul 17 14:40:51.281677 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.281295    2575 scope.go:117] "RemoveContainer" containerID="b82dbb6297469b0acd1c69aafea9e8cbfda52d650bc2e0b51801474804894cb0"
Jul 17 14:40:51.304623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.304598    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"]
Jul 17 14:40:51.308412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.308389    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"]
Jul 17 14:40:51.314989 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.314965    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-2ndsj"]
Jul 17 14:40:51.440646 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.440620    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"
Jul 17 14:40:51.503860 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.503829    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83da60ca-94d1-491a-b2cb-b4db1e5559f1-maas-api-service-ca\") pod \"83da60ca-94d1-491a-b2cb-b4db1e5559f1\" (UID: \"83da60ca-94d1-491a-b2cb-b4db1e5559f1\") "
Jul 17 14:40:51.504156 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.504136    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83da60ca-94d1-491a-b2cb-b4db1e5559f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83da60ca-94d1-491a-b2cb-b4db1e5559f1" (UID: "83da60ca-94d1-491a-b2cb-b4db1e5559f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:40:51.604995 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:51.604936    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83da60ca-94d1-491a-b2cb-b4db1e5559f1-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:40:52.075421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:52.075396    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" path="/var/lib/kubelet/pods/83ece7f9-d23b-4666-b0a3-47c4c6d25977/volumes"
Jul 17 14:40:52.289515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:52.289490    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"
Jul 17 14:40:52.289515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:52.289498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b" event={"ID":"83da60ca-94d1-491a-b2cb-b4db1e5559f1","Type":"ContainerDied","Data":"739d7191c5e7d079d44751ec9f9a589c47d40efc241888b8b3fd321e670cd7d8"}
Jul 17 14:40:52.289958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:52.289562    2575 scope.go:117] "RemoveContainer" containerID="9f2f481bfd998efeca3a85c91737d64d64c144768a1875ce1f0893bf71ade234"
Jul 17 14:40:52.305691 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:52.305671    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"]
Jul 17 14:40:52.310117 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:52.310096    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-jzt5b"]
Jul 17 14:40:54.074568 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:40:54.074536    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" path="/var/lib/kubelet/pods/83da60ca-94d1-491a-b2cb-b4db1e5559f1/volumes"
Jul 17 14:41:01.068844 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:01.068817    2575 scope.go:117] "RemoveContainer" containerID="19bdb7653545dc1de626f866873941ab141c7d71fd56fc8f25101101a83d266b"
Jul 17 14:41:02.346803 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:02.346775    2575 generic.go:356] "Generic (PLEG): container finished" podID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerID="ec35ec6af059fa5ce01e2d049c978dbb8fcce9037ee4e1eb06b7480ebd9f5fcd" exitCode=6
Jul 17 14:41:02.347168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:02.346843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" event={"ID":"1da71a8b-e839-499f-b1b4-b7075d5c1958","Type":"ContainerDied","Data":"ec35ec6af059fa5ce01e2d049c978dbb8fcce9037ee4e1eb06b7480ebd9f5fcd"}
Jul 17 14:41:02.347168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:02.346900    2575 scope.go:117] "RemoveContainer" containerID="19bdb7653545dc1de626f866873941ab141c7d71fd56fc8f25101101a83d266b"
Jul 17 14:41:02.347168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:02.347158    2575 scope.go:117] "RemoveContainer" containerID="ec35ec6af059fa5ce01e2d049c978dbb8fcce9037ee4e1eb06b7480ebd9f5fcd"
Jul 17 14:41:02.347373 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:02.347355    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz_odh-ai-gateway-infra(1da71a8b-e839-499f-b1b4-b7075d5c1958)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958"
Jul 17 14:41:03.373162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:03.373134    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"]
Jul 17 14:41:03.504525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:03.504500    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"
Jul 17 14:41:03.594790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:03.594758    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da71a8b-e839-499f-b1b4-b7075d5c1958-maas-api-service-ca\") pod \"1da71a8b-e839-499f-b1b4-b7075d5c1958\" (UID: \"1da71a8b-e839-499f-b1b4-b7075d5c1958\") "
Jul 17 14:41:03.595100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:03.595079    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da71a8b-e839-499f-b1b4-b7075d5c1958-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1da71a8b-e839-499f-b1b4-b7075d5c1958" (UID: "1da71a8b-e839-499f-b1b4-b7075d5c1958"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:41:03.695968 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:03.695906    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da71a8b-e839-499f-b1b4-b7075d5c1958-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:41:04.358887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:04.358814    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"
Jul 17 14:41:04.358887 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:04.358861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz" event={"ID":"1da71a8b-e839-499f-b1b4-b7075d5c1958","Type":"ContainerDied","Data":"89d6efb58d87995d0d2eb31e0daeb8967515f8ec1d1251310a5061b4fb1cab22"}
Jul 17 14:41:04.359091 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:04.358927    2575 scope.go:117] "RemoveContainer" containerID="ec35ec6af059fa5ce01e2d049c978dbb8fcce9037ee4e1eb06b7480ebd9f5fcd"
Jul 17 14:41:04.374978 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:04.374955    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"]
Jul 17 14:41:04.378956 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:04.378933    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ppqbz"]
Jul 17 14:41:06.075881 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:06.075845    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" path="/var/lib/kubelet/pods/1da71a8b-e839-499f-b1b4-b7075d5c1958/volumes"
Jul 17 14:41:12.302016 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.301977    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"]
Jul 17 14:41:12.302559 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302540    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302562    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302575    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302583    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302595    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302604    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302616    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302625    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:12.302649 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302649    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302657    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302671    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302679    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302688    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302696    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302715    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302724    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302736    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302744    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302832    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302845    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e82de0c-557e-4be6-8518-4a13f9824acd" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302854    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302864    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302876    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302883    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302890    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302897    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.303022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.302904    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da71a8b-e839-499f-b1b4-b7075d5c1958" containerName="revoke-keys"
Jul 17 14:41:12.306427 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.306405    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"
Jul 17 14:41:12.308686 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.308667    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:41:12.313705 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.313341    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"]
Jul 17 14:41:12.363779 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.363751    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-djm56\" (UID: \"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"
Jul 17 14:41:12.464838 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.464814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-djm56\" (UID: \"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"
Jul 17 14:41:12.465412 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.465393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-djm56\" (UID: \"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"
Jul 17 14:41:12.618634 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.618584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"
Jul 17 14:41:12.752802 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:12.752774    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"]
Jul 17 14:41:12.754345 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:41:12.754315    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f73302_7c8c_42bf_9ba1_3e7dec6cbc17.slice/crio-effd891d1f8abe4d96939cd53bec0b89fbd0e79fc61dd197929d7e3a98272306 WatchSource:0}: Error finding container effd891d1f8abe4d96939cd53bec0b89fbd0e79fc61dd197929d7e3a98272306: Status 404 returned error can't find the container with id effd891d1f8abe4d96939cd53bec0b89fbd0e79fc61dd197929d7e3a98272306
Jul 17 14:41:13.405658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:13.405632    2575 generic.go:356] "Generic (PLEG): container finished" podID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerID="0120991149ec970741a4fe9d5b9fb169528e3d7c9f93efb937751b6ca35010c1" exitCode=6
Jul 17 14:41:13.406014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:13.405718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" event={"ID":"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17","Type":"ContainerDied","Data":"0120991149ec970741a4fe9d5b9fb169528e3d7c9f93efb937751b6ca35010c1"}
Jul 17 14:41:13.406014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:13.405776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" event={"ID":"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17","Type":"ContainerStarted","Data":"effd891d1f8abe4d96939cd53bec0b89fbd0e79fc61dd197929d7e3a98272306"}
Jul 17 14:41:13.406014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:13.405871    2575 scope.go:117] "RemoveContainer" containerID="0120991149ec970741a4fe9d5b9fb169528e3d7c9f93efb937751b6ca35010c1"
Jul 17 14:41:14.411999 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:14.411972    2575 generic.go:356] "Generic (PLEG): container finished" podID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerID="ce3345e5a633af3f13e4ca03b2b74e1e586b2601b2cee3137ce9f80acd5c5791" exitCode=6
Jul 17 14:41:14.412440 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:14.412053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" event={"ID":"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17","Type":"ContainerDied","Data":"ce3345e5a633af3f13e4ca03b2b74e1e586b2601b2cee3137ce9f80acd5c5791"}
Jul 17 14:41:14.412440 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:14.412123    2575 scope.go:117] "RemoveContainer" containerID="0120991149ec970741a4fe9d5b9fb169528e3d7c9f93efb937751b6ca35010c1"
Jul 17 14:41:14.412440 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:14.412190    2575 scope.go:117] "RemoveContainer" containerID="ce3345e5a633af3f13e4ca03b2b74e1e586b2601b2cee3137ce9f80acd5c5791"
Jul 17 14:41:14.412440 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:14.412369    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-djm56_odh-ai-gateway-infra(d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17"
Jul 17 14:41:15.418928 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:15.418903    2575 scope.go:117] "RemoveContainer" containerID="ce3345e5a633af3f13e4ca03b2b74e1e586b2601b2cee3137ce9f80acd5c5791"
Jul 17 14:41:15.419311 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:15.419072    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-djm56_odh-ai-gateway-infra(d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17"
Jul 17 14:41:23.912380 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:23.912345    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"]
Jul 17 14:41:23.912744 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:23.912730    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:23.912744 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:23.912744    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ece7f9-d23b-4666-b0a3-47c4c6d25977" containerName="revoke-keys"
Jul 17 14:41:23.912829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:23.912820    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83da60ca-94d1-491a-b2cb-b4db1e5559f1" containerName="revoke-keys"
Jul 17 14:41:23.917386 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:23.917370    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"
Jul 17 14:41:23.924671 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:23.924647    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"]
Jul 17 14:41:24.055911 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:24.055881    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14d0495-60d0-4245-9c72-ed7f6a3e28cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8\" (UID: \"e14d0495-60d0-4245-9c72-ed7f6a3e28cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"
Jul 17 14:41:24.157082 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:24.157046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14d0495-60d0-4245-9c72-ed7f6a3e28cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8\" (UID: \"e14d0495-60d0-4245-9c72-ed7f6a3e28cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"
Jul 17 14:41:24.157915 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:24.157891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14d0495-60d0-4245-9c72-ed7f6a3e28cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8\" (UID: \"e14d0495-60d0-4245-9c72-ed7f6a3e28cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"
Jul 17 14:41:24.228045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:24.228028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"
Jul 17 14:41:24.356349 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:24.356326    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"]
Jul 17 14:41:24.358957 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:41:24.358923    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14d0495_60d0_4245_9c72_ed7f6a3e28cb.slice/crio-7722b7f43e9e94448892ac56712e84c9e55216e310b330519347c422cbae4a8d WatchSource:0}: Error finding container 7722b7f43e9e94448892ac56712e84c9e55216e310b330519347c422cbae4a8d: Status 404 returned error can't find the container with id 7722b7f43e9e94448892ac56712e84c9e55216e310b330519347c422cbae4a8d
Jul 17 14:41:24.465269 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:24.465240    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" event={"ID":"e14d0495-60d0-4245-9c72-ed7f6a3e28cb","Type":"ContainerStarted","Data":"7722b7f43e9e94448892ac56712e84c9e55216e310b330519347c422cbae4a8d"}
Jul 17 14:41:25.471678 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:25.471647    2575 generic.go:356] "Generic (PLEG): container finished" podID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerID="5e04b9d45efd68e92a7b66f49aed5914ab8c7730595e7687ea65cd4e899b38be" exitCode=6
Jul 17 14:41:25.472050 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:25.471724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" event={"ID":"e14d0495-60d0-4245-9c72-ed7f6a3e28cb","Type":"ContainerDied","Data":"5e04b9d45efd68e92a7b66f49aed5914ab8c7730595e7687ea65cd4e899b38be"}
Jul 17 14:41:25.472050 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:25.471881    2575 scope.go:117] "RemoveContainer" containerID="5e04b9d45efd68e92a7b66f49aed5914ab8c7730595e7687ea65cd4e899b38be"
Jul 17 14:41:26.069206 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:26.069180    2575 scope.go:117] "RemoveContainer" containerID="ce3345e5a633af3f13e4ca03b2b74e1e586b2601b2cee3137ce9f80acd5c5791"
Jul 17 14:41:26.479105 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:26.479073    2575 generic.go:356] "Generic (PLEG): container finished" podID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerID="d83a73fbf3c0d7c8aabae35f3765ee3685b6c04be16a0430e7c2f46388eb7daf" exitCode=6
Jul 17 14:41:26.479488 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:26.479159    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" event={"ID":"e14d0495-60d0-4245-9c72-ed7f6a3e28cb","Type":"ContainerDied","Data":"d83a73fbf3c0d7c8aabae35f3765ee3685b6c04be16a0430e7c2f46388eb7daf"}
Jul 17 14:41:26.479488 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:26.479215    2575 scope.go:117] "RemoveContainer" containerID="5e04b9d45efd68e92a7b66f49aed5914ab8c7730595e7687ea65cd4e899b38be"
Jul 17 14:41:26.479488 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:26.479359    2575 scope.go:117] "RemoveContainer" containerID="d83a73fbf3c0d7c8aabae35f3765ee3685b6c04be16a0430e7c2f46388eb7daf"
Jul 17 14:41:26.479626 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:26.479543    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8_odh-ai-gateway-infra(e14d0495-60d0-4245-9c72-ed7f6a3e28cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb"
Jul 17 14:41:26.481198 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:26.481178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" event={"ID":"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17","Type":"ContainerStarted","Data":"7f1a4697a644b9fa6e0d36771b21dfbb53ab3c726293b7ae6c1643d8d56eb170"}
Jul 17 14:41:26.520349 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:26.516711    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" podStartSLOduration=14.516670149 podStartE2EDuration="14.516670149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:41:26.512509226 +0000 UTC m=+3364.899856667" watchObservedRunningTime="2026-07-17 14:41:26.516670149 +0000 UTC m=+3364.904017590"
Jul 17 14:41:27.488686 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:27.488656    2575 generic.go:356] "Generic (PLEG): container finished" podID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerID="7f1a4697a644b9fa6e0d36771b21dfbb53ab3c726293b7ae6c1643d8d56eb170" exitCode=6
Jul 17 14:41:27.489075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:27.488729    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" event={"ID":"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17","Type":"ContainerDied","Data":"7f1a4697a644b9fa6e0d36771b21dfbb53ab3c726293b7ae6c1643d8d56eb170"}
Jul 17 14:41:27.489075 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:27.488819    2575 scope.go:117] "RemoveContainer" containerID="ce3345e5a633af3f13e4ca03b2b74e1e586b2601b2cee3137ce9f80acd5c5791"
Jul 17 14:41:27.489198 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:27.489082    2575 scope.go:117] "RemoveContainer" containerID="7f1a4697a644b9fa6e0d36771b21dfbb53ab3c726293b7ae6c1643d8d56eb170"
Jul 17 14:41:27.489273 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:27.489255    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-djm56_odh-ai-gateway-infra(d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17"
Jul 17 14:41:27.490564 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:27.490548    2575 scope.go:117] "RemoveContainer" containerID="d83a73fbf3c0d7c8aabae35f3765ee3685b6c04be16a0430e7c2f46388eb7daf"
Jul 17 14:41:27.490695 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:27.490679    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8_odh-ai-gateway-infra(e14d0495-60d0-4245-9c72-ed7f6a3e28cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb"
Jul 17 14:41:27.523590 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:27.523564    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"]
Jul 17 14:41:28.640916 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:28.640892    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"
Jul 17 14:41:28.797395 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:28.797326    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17-maas-api-service-ca\") pod \"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17\" (UID: \"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17\") "
Jul 17 14:41:28.797646 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:28.797627    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" (UID: "d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:41:28.897837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:28.897814    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:41:29.503348 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.503325    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"
Jul 17 14:41:29.503531 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.503349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56" event={"ID":"d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17","Type":"ContainerDied","Data":"effd891d1f8abe4d96939cd53bec0b89fbd0e79fc61dd197929d7e3a98272306"}
Jul 17 14:41:29.503531 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.503407    2575 scope.go:117] "RemoveContainer" containerID="7f1a4697a644b9fa6e0d36771b21dfbb53ab3c726293b7ae6c1643d8d56eb170"
Jul 17 14:41:29.526445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.526424    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"]
Jul 17 14:41:29.531398 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.531379    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-djm56"]
Jul 17 14:41:29.874888 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.874815    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"]
Jul 17 14:41:29.875222 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875209    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.875268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875223    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.875268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875231    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.875268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875236    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.875268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875248    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.875268 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875253    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.875454 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875342    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.875454 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.875349    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:29.879664 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.879648    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"
Jul 17 14:41:29.887475 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:29.887304    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"]
Jul 17 14:41:30.007123 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:30.007090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ef3a8-683b-4223-9dc0-ddd32be2d11e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5\" (UID: \"312ef3a8-683b-4223-9dc0-ddd32be2d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"
Jul 17 14:41:30.074947 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:30.074920    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" path="/var/lib/kubelet/pods/d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17/volumes"
Jul 17 14:41:30.107831 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:30.107802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ef3a8-683b-4223-9dc0-ddd32be2d11e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5\" (UID: \"312ef3a8-683b-4223-9dc0-ddd32be2d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"
Jul 17 14:41:30.108429 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:30.108412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ef3a8-683b-4223-9dc0-ddd32be2d11e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5\" (UID: \"312ef3a8-683b-4223-9dc0-ddd32be2d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"
Jul 17 14:41:30.191829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:30.191773    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"
Jul 17 14:41:30.324394 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:30.324367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"]
Jul 17 14:41:30.326212 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:41:30.326180    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312ef3a8_683b_4223_9dc0_ddd32be2d11e.slice/crio-aeaa26432d04c4278f4f4c067f7faa2cbdaa0c1227e6a9be203976b4d9f32ee6 WatchSource:0}: Error finding container aeaa26432d04c4278f4f4c067f7faa2cbdaa0c1227e6a9be203976b4d9f32ee6: Status 404 returned error can't find the container with id aeaa26432d04c4278f4f4c067f7faa2cbdaa0c1227e6a9be203976b4d9f32ee6
Jul 17 14:41:30.510313 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:30.510263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" event={"ID":"312ef3a8-683b-4223-9dc0-ddd32be2d11e","Type":"ContainerStarted","Data":"aeaa26432d04c4278f4f4c067f7faa2cbdaa0c1227e6a9be203976b4d9f32ee6"}
Jul 17 14:41:31.516880 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:31.516846    2575 generic.go:356] "Generic (PLEG): container finished" podID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerID="201cc34c4ed3d1cd8edf398b486b0e7824c5c696cd703cc1d6418dca4a910cab" exitCode=6
Jul 17 14:41:31.517317 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:31.516949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" event={"ID":"312ef3a8-683b-4223-9dc0-ddd32be2d11e","Type":"ContainerDied","Data":"201cc34c4ed3d1cd8edf398b486b0e7824c5c696cd703cc1d6418dca4a910cab"}
Jul 17 14:41:31.517317 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:31.517073    2575 scope.go:117] "RemoveContainer" containerID="201cc34c4ed3d1cd8edf398b486b0e7824c5c696cd703cc1d6418dca4a910cab"
Jul 17 14:41:32.524296 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:32.524251    2575 generic.go:356] "Generic (PLEG): container finished" podID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerID="896d2565ec9a00013e9b61dcb3e0d0ed9883494aae76c695e0bd74250ace6eac" exitCode=6
Jul 17 14:41:32.524723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:32.524311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" event={"ID":"312ef3a8-683b-4223-9dc0-ddd32be2d11e","Type":"ContainerDied","Data":"896d2565ec9a00013e9b61dcb3e0d0ed9883494aae76c695e0bd74250ace6eac"}
Jul 17 14:41:32.524723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:32.524369    2575 scope.go:117] "RemoveContainer" containerID="201cc34c4ed3d1cd8edf398b486b0e7824c5c696cd703cc1d6418dca4a910cab"
Jul 17 14:41:32.524723 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:32.524535    2575 scope.go:117] "RemoveContainer" containerID="896d2565ec9a00013e9b61dcb3e0d0ed9883494aae76c695e0bd74250ace6eac"
Jul 17 14:41:32.524832 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:32.524723    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5_odh-ai-gateway-infra(312ef3a8-683b-4223-9dc0-ddd32be2d11e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e"
Jul 17 14:41:33.531863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:33.531837    2575 scope.go:117] "RemoveContainer" containerID="896d2565ec9a00013e9b61dcb3e0d0ed9883494aae76c695e0bd74250ace6eac"
Jul 17 14:41:33.532199 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:33.532020    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5_odh-ai-gateway-infra(312ef3a8-683b-4223-9dc0-ddd32be2d11e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e"
Jul 17 14:41:39.068849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:39.068817    2575 scope.go:117] "RemoveContainer" containerID="d83a73fbf3c0d7c8aabae35f3765ee3685b6c04be16a0430e7c2f46388eb7daf"
Jul 17 14:41:39.563344 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:39.563314    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" event={"ID":"e14d0495-60d0-4245-9c72-ed7f6a3e28cb","Type":"ContainerStarted","Data":"e00b2d1670e7241d6ccf7a5eda828810d2a6663ab6ebdc4da89e5b4904fd9b7c"}
Jul 17 14:41:39.584787 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:39.584745    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" podStartSLOduration=16.584729901 podStartE2EDuration="16.584729901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:41:39.583693151 +0000 UTC m=+3377.971040592" watchObservedRunningTime="2026-07-17 14:41:39.584729901 +0000 UTC m=+3377.972077340"
Jul 17 14:41:40.570488 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:40.570457    2575 generic.go:356] "Generic (PLEG): container finished" podID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerID="e00b2d1670e7241d6ccf7a5eda828810d2a6663ab6ebdc4da89e5b4904fd9b7c" exitCode=6
Jul 17 14:41:40.570876 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:40.570527    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" event={"ID":"e14d0495-60d0-4245-9c72-ed7f6a3e28cb","Type":"ContainerDied","Data":"e00b2d1670e7241d6ccf7a5eda828810d2a6663ab6ebdc4da89e5b4904fd9b7c"}
Jul 17 14:41:40.570876 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:40.570592    2575 scope.go:117] "RemoveContainer" containerID="d83a73fbf3c0d7c8aabae35f3765ee3685b6c04be16a0430e7c2f46388eb7daf"
Jul 17 14:41:40.570999 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:40.570888    2575 scope.go:117] "RemoveContainer" containerID="e00b2d1670e7241d6ccf7a5eda828810d2a6663ab6ebdc4da89e5b4904fd9b7c"
Jul 17 14:41:40.571089 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:40.571069    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8_odh-ai-gateway-infra(e14d0495-60d0-4245-9c72-ed7f6a3e28cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb"
Jul 17 14:41:40.595073 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:40.595051    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"]
Jul 17 14:41:41.707815 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:41.707791    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"
Jul 17 14:41:41.799478 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:41.799452    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14d0495-60d0-4245-9c72-ed7f6a3e28cb-maas-api-service-ca\") pod \"e14d0495-60d0-4245-9c72-ed7f6a3e28cb\" (UID: \"e14d0495-60d0-4245-9c72-ed7f6a3e28cb\") "
Jul 17 14:41:41.799796 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:41.799773    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14d0495-60d0-4245-9c72-ed7f6a3e28cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e14d0495-60d0-4245-9c72-ed7f6a3e28cb" (UID: "e14d0495-60d0-4245-9c72-ed7f6a3e28cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:41:41.900546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:41.900491    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14d0495-60d0-4245-9c72-ed7f6a3e28cb-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:41:42.583600 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:42.583570    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"
Jul 17 14:41:42.583758 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:42.583573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8" event={"ID":"e14d0495-60d0-4245-9c72-ed7f6a3e28cb","Type":"ContainerDied","Data":"7722b7f43e9e94448892ac56712e84c9e55216e310b330519347c422cbae4a8d"}
Jul 17 14:41:42.583758 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:42.583731    2575 scope.go:117] "RemoveContainer" containerID="e00b2d1670e7241d6ccf7a5eda828810d2a6663ab6ebdc4da89e5b4904fd9b7c"
Jul 17 14:41:42.600219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:42.600191    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"]
Jul 17 14:41:42.608559 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:42.608538    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-2l8z8"]
Jul 17 14:41:43.060032 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060003    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"]
Jul 17 14:41:43.060434 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060420    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:41:43.060479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060436    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:41:43.060479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060455    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:41:43.060479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060460    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:41:43.060570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060525    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:41:43.060570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060533    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f73302-7c8c-42bf-9ba1-3e7dec6cbc17" containerName="revoke-keys"
Jul 17 14:41:43.060570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.060541    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:41:43.063212 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.063197    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"
Jul 17 14:41:43.073448 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.073428    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"]
Jul 17 14:41:43.215001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.214977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/887e5cae-de00-40e3-8e62-03a9521b4c15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm\" (UID: \"887e5cae-de00-40e3-8e62-03a9521b4c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"
Jul 17 14:41:43.316425 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.316361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/887e5cae-de00-40e3-8e62-03a9521b4c15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm\" (UID: \"887e5cae-de00-40e3-8e62-03a9521b4c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"
Jul 17 14:41:43.316909 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.316890    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/887e5cae-de00-40e3-8e62-03a9521b4c15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm\" (UID: \"887e5cae-de00-40e3-8e62-03a9521b4c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"
Jul 17 14:41:43.373644 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.373622    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"
Jul 17 14:41:43.501444 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.501415    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"]
Jul 17 14:41:43.501577 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:41:43.501526    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887e5cae_de00_40e3_8e62_03a9521b4c15.slice/crio-98f09c7de7e278509ad68829b854837d78ad9befbb821b000d5f7f9fac783d70 WatchSource:0}: Error finding container 98f09c7de7e278509ad68829b854837d78ad9befbb821b000d5f7f9fac783d70: Status 404 returned error can't find the container with id 98f09c7de7e278509ad68829b854837d78ad9befbb821b000d5f7f9fac783d70
Jul 17 14:41:43.589315 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:43.589231    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" event={"ID":"887e5cae-de00-40e3-8e62-03a9521b4c15","Type":"ContainerStarted","Data":"98f09c7de7e278509ad68829b854837d78ad9befbb821b000d5f7f9fac783d70"}
Jul 17 14:41:44.075630 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:44.075601    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" path="/var/lib/kubelet/pods/e14d0495-60d0-4245-9c72-ed7f6a3e28cb/volumes"
Jul 17 14:41:44.596394 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:44.596367    2575 generic.go:356] "Generic (PLEG): container finished" podID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerID="09e02d9fd7e2b9b118343ba9b68e037323da6c172f6e47bbfb6450cbaaf3dcef" exitCode=6
Jul 17 14:41:44.596623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:44.596449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" event={"ID":"887e5cae-de00-40e3-8e62-03a9521b4c15","Type":"ContainerDied","Data":"09e02d9fd7e2b9b118343ba9b68e037323da6c172f6e47bbfb6450cbaaf3dcef"}
Jul 17 14:41:44.596623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:44.596600    2575 scope.go:117] "RemoveContainer" containerID="09e02d9fd7e2b9b118343ba9b68e037323da6c172f6e47bbfb6450cbaaf3dcef"
Jul 17 14:41:45.602837 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:45.602803    2575 generic.go:356] "Generic (PLEG): container finished" podID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerID="f31d5affcd942abd667f14932cee156bf8f0b54aed9c3a55a892ff2d5b7e4986" exitCode=6
Jul 17 14:41:45.603244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:45.602880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" event={"ID":"887e5cae-de00-40e3-8e62-03a9521b4c15","Type":"ContainerDied","Data":"f31d5affcd942abd667f14932cee156bf8f0b54aed9c3a55a892ff2d5b7e4986"}
Jul 17 14:41:45.603244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:45.602934    2575 scope.go:117] "RemoveContainer" containerID="09e02d9fd7e2b9b118343ba9b68e037323da6c172f6e47bbfb6450cbaaf3dcef"
Jul 17 14:41:45.603244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:45.603144    2575 scope.go:117] "RemoveContainer" containerID="f31d5affcd942abd667f14932cee156bf8f0b54aed9c3a55a892ff2d5b7e4986"
Jul 17 14:41:45.603444 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:45.603364    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm_odh-ai-gateway-infra(887e5cae-de00-40e3-8e62-03a9521b4c15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15"
Jul 17 14:41:46.609854 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:46.609830    2575 scope.go:117] "RemoveContainer" containerID="f31d5affcd942abd667f14932cee156bf8f0b54aed9c3a55a892ff2d5b7e4986"
Jul 17 14:41:46.610206 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:46.609988    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm_odh-ai-gateway-infra(887e5cae-de00-40e3-8e62-03a9521b4c15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15"
Jul 17 14:41:49.069581 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:49.069556    2575 scope.go:117] "RemoveContainer" containerID="896d2565ec9a00013e9b61dcb3e0d0ed9883494aae76c695e0bd74250ace6eac"
Jul 17 14:41:49.633063 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:49.633027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" event={"ID":"312ef3a8-683b-4223-9dc0-ddd32be2d11e","Type":"ContainerStarted","Data":"e9865288e297e2ec71e4c0a66f22f597e633f0dbd13bb761b35ff9a799724d77"}
Jul 17 14:41:49.652841 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:49.652798    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" podStartSLOduration=20.652785718 podStartE2EDuration="20.652785718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:41:49.651210389 +0000 UTC m=+3388.038557828" watchObservedRunningTime="2026-07-17 14:41:49.652785718 +0000 UTC m=+3388.040133160"
Jul 17 14:41:50.640167 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:50.640135    2575 generic.go:356] "Generic (PLEG): container finished" podID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerID="e9865288e297e2ec71e4c0a66f22f597e633f0dbd13bb761b35ff9a799724d77" exitCode=6
Jul 17 14:41:50.640610 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:50.640206    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" event={"ID":"312ef3a8-683b-4223-9dc0-ddd32be2d11e","Type":"ContainerDied","Data":"e9865288e297e2ec71e4c0a66f22f597e633f0dbd13bb761b35ff9a799724d77"}
Jul 17 14:41:50.640610 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:50.640269    2575 scope.go:117] "RemoveContainer" containerID="896d2565ec9a00013e9b61dcb3e0d0ed9883494aae76c695e0bd74250ace6eac"
Jul 17 14:41:50.640610 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:50.640536    2575 scope.go:117] "RemoveContainer" containerID="e9865288e297e2ec71e4c0a66f22f597e633f0dbd13bb761b35ff9a799724d77"
Jul 17 14:41:50.640749 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:50.640732    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5_odh-ai-gateway-infra(312ef3a8-683b-4223-9dc0-ddd32be2d11e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e"
Jul 17 14:41:50.670784 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:50.670756    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"]
Jul 17 14:41:51.774229 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:51.774207    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"
Jul 17 14:41:51.883400 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:51.883374    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ef3a8-683b-4223-9dc0-ddd32be2d11e-maas-api-service-ca\") pod \"312ef3a8-683b-4223-9dc0-ddd32be2d11e\" (UID: \"312ef3a8-683b-4223-9dc0-ddd32be2d11e\") "
Jul 17 14:41:51.883737 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:51.883718    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312ef3a8-683b-4223-9dc0-ddd32be2d11e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "312ef3a8-683b-4223-9dc0-ddd32be2d11e" (UID: "312ef3a8-683b-4223-9dc0-ddd32be2d11e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:41:51.983940 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:51.983918    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ef3a8-683b-4223-9dc0-ddd32be2d11e-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:41:52.655418 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:52.655391    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"
Jul 17 14:41:52.655745 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:52.655427    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5" event={"ID":"312ef3a8-683b-4223-9dc0-ddd32be2d11e","Type":"ContainerDied","Data":"aeaa26432d04c4278f4f4c067f7faa2cbdaa0c1227e6a9be203976b4d9f32ee6"}
Jul 17 14:41:52.655745 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:52.655485    2575 scope.go:117] "RemoveContainer" containerID="e9865288e297e2ec71e4c0a66f22f597e633f0dbd13bb761b35ff9a799724d77"
Jul 17 14:41:52.675971 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:52.675948    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"]
Jul 17 14:41:52.679458 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:52.679437    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-fh7s5"]
Jul 17 14:41:54.074757 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:54.074723    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" path="/var/lib/kubelet/pods/312ef3a8-683b-4223-9dc0-ddd32be2d11e/volumes"
Jul 17 14:41:58.069204 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:58.069167    2575 scope.go:117] "RemoveContainer" containerID="f31d5affcd942abd667f14932cee156bf8f0b54aed9c3a55a892ff2d5b7e4986"
Jul 17 14:41:58.686324 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:58.686299    2575 generic.go:356] "Generic (PLEG): container finished" podID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerID="2b9487095a054fdb930f86ff086cafbc8fc9394cb838a479bf4edda74e927134" exitCode=6
Jul 17 14:41:58.686451 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:58.686372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" event={"ID":"887e5cae-de00-40e3-8e62-03a9521b4c15","Type":"ContainerDied","Data":"2b9487095a054fdb930f86ff086cafbc8fc9394cb838a479bf4edda74e927134"}
Jul 17 14:41:58.686512 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:58.686457    2575 scope.go:117] "RemoveContainer" containerID="f31d5affcd942abd667f14932cee156bf8f0b54aed9c3a55a892ff2d5b7e4986"
Jul 17 14:41:58.686696 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:58.686675    2575 scope.go:117] "RemoveContainer" containerID="2b9487095a054fdb930f86ff086cafbc8fc9394cb838a479bf4edda74e927134"
Jul 17 14:41:58.686919 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:41:58.686894    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm_odh-ai-gateway-infra(887e5cae-de00-40e3-8e62-03a9521b4c15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15"
Jul 17 14:41:59.722948 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:59.722921    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"]
Jul 17 14:41:59.856382 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:59.856358    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"
Jul 17 14:41:59.942635 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:59.942607    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/887e5cae-de00-40e3-8e62-03a9521b4c15-maas-api-service-ca\") pod \"887e5cae-de00-40e3-8e62-03a9521b4c15\" (UID: \"887e5cae-de00-40e3-8e62-03a9521b4c15\") "
Jul 17 14:41:59.942912 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:41:59.942892    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887e5cae-de00-40e3-8e62-03a9521b4c15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "887e5cae-de00-40e3-8e62-03a9521b4c15" (UID: "887e5cae-de00-40e3-8e62-03a9521b4c15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:42:00.043722 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:00.043657    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/887e5cae-de00-40e3-8e62-03a9521b4c15-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:42:00.699518 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:00.699487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm" event={"ID":"887e5cae-de00-40e3-8e62-03a9521b4c15","Type":"ContainerDied","Data":"98f09c7de7e278509ad68829b854837d78ad9befbb821b000d5f7f9fac783d70"}
Jul 17 14:42:00.699518 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:00.699515    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"
Jul 17 14:42:00.699757 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:00.699543    2575 scope.go:117] "RemoveContainer" containerID="2b9487095a054fdb930f86ff086cafbc8fc9394cb838a479bf4edda74e927134"
Jul 17 14:42:00.716560 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:00.716534    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"]
Jul 17 14:42:00.721131 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:00.721106    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-srvwm"]
Jul 17 14:42:02.074325 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.074272    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" path="/var/lib/kubelet/pods/887e5cae-de00-40e3-8e62-03a9521b4c15/volumes"
Jul 17 14:42:02.343298 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343198    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"]
Jul 17 14:42:02.343680 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343661    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343682    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343693    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343701    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343715    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343724    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343753    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343761    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:42:02.343775 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343774    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:02.344143 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343784    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:02.344143 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343889    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:02.344143 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343903    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:02.344143 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343913    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:02.344143 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343921    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14d0495-60d0-4245-9c72-ed7f6a3e28cb" containerName="revoke-keys"
Jul 17 14:42:02.344143 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.343936    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:02.347406 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.347388    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"
Jul 17 14:42:02.349573 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.349554    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:42:02.353934 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.353911    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"]
Jul 17 14:42:02.362590 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.362558    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8948d36-48a9-4844-9580-b3734ec2c235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-d2gnx\" (UID: \"e8948d36-48a9-4844-9580-b3734ec2c235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"
Jul 17 14:42:02.463421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.463398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8948d36-48a9-4844-9580-b3734ec2c235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-d2gnx\" (UID: \"e8948d36-48a9-4844-9580-b3734ec2c235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"
Jul 17 14:42:02.463985 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.463964    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8948d36-48a9-4844-9580-b3734ec2c235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-d2gnx\" (UID: \"e8948d36-48a9-4844-9580-b3734ec2c235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"
Jul 17 14:42:02.658898 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.658843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"
Jul 17 14:42:02.788131 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:02.788108    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"]
Jul 17 14:42:02.790529 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:42:02.790499    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8948d36_48a9_4844_9580_b3734ec2c235.slice/crio-0e9ed9089d6ae188e15aa76650bd41a9dddf268643be641ce05fdbe0b5e5a70e WatchSource:0}: Error finding container 0e9ed9089d6ae188e15aa76650bd41a9dddf268643be641ce05fdbe0b5e5a70e: Status 404 returned error can't find the container with id 0e9ed9089d6ae188e15aa76650bd41a9dddf268643be641ce05fdbe0b5e5a70e
Jul 17 14:42:03.719189 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:03.719161    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8948d36-48a9-4844-9580-b3734ec2c235" containerID="c2aac51fa13732bfe3328cfe35a4f01e39395c27063cfbbea6146603094ccbc4" exitCode=6
Jul 17 14:42:03.719550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:03.719233    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" event={"ID":"e8948d36-48a9-4844-9580-b3734ec2c235","Type":"ContainerDied","Data":"c2aac51fa13732bfe3328cfe35a4f01e39395c27063cfbbea6146603094ccbc4"}
Jul 17 14:42:03.719550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:03.719316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" event={"ID":"e8948d36-48a9-4844-9580-b3734ec2c235","Type":"ContainerStarted","Data":"0e9ed9089d6ae188e15aa76650bd41a9dddf268643be641ce05fdbe0b5e5a70e"}
Jul 17 14:42:03.719550 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:03.719445    2575 scope.go:117] "RemoveContainer" containerID="c2aac51fa13732bfe3328cfe35a4f01e39395c27063cfbbea6146603094ccbc4"
Jul 17 14:42:04.726792 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:04.726763    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8948d36-48a9-4844-9580-b3734ec2c235" containerID="14fecc6bac9aa16e3be3e6b59e9cd6937f0cfed9c69c80e4115a5a127af8d210" exitCode=6
Jul 17 14:42:04.727160 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:04.726849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" event={"ID":"e8948d36-48a9-4844-9580-b3734ec2c235","Type":"ContainerDied","Data":"14fecc6bac9aa16e3be3e6b59e9cd6937f0cfed9c69c80e4115a5a127af8d210"}
Jul 17 14:42:04.727160 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:04.726900    2575 scope.go:117] "RemoveContainer" containerID="c2aac51fa13732bfe3328cfe35a4f01e39395c27063cfbbea6146603094ccbc4"
Jul 17 14:42:04.727160 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:04.726999    2575 scope.go:117] "RemoveContainer" containerID="14fecc6bac9aa16e3be3e6b59e9cd6937f0cfed9c69c80e4115a5a127af8d210"
Jul 17 14:42:04.727309 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:04.727213    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-d2gnx_odh-ai-gateway-infra(e8948d36-48a9-4844-9580-b3734ec2c235)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" podUID="e8948d36-48a9-4844-9580-b3734ec2c235"
Jul 17 14:42:05.733056 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:05.733023    2575 scope.go:117] "RemoveContainer" containerID="14fecc6bac9aa16e3be3e6b59e9cd6937f0cfed9c69c80e4115a5a127af8d210"
Jul 17 14:42:05.733448 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:05.733188    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-d2gnx_odh-ai-gateway-infra(e8948d36-48a9-4844-9580-b3734ec2c235)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" podUID="e8948d36-48a9-4844-9580-b3734ec2c235"
Jul 17 14:42:13.979445 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.979412    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"]
Jul 17 14:42:13.979817 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.979803    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:13.979817 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.979818    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:13.979893 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.979831    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:13.979893 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.979837    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:13.979954 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.979920    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="312ef3a8-683b-4223-9dc0-ddd32be2d11e" containerName="revoke-keys"
Jul 17 14:42:13.979954 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.979927    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="887e5cae-de00-40e3-8e62-03a9521b4c15" containerName="revoke-keys"
Jul 17 14:42:13.983014 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.982994    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"
Jul 17 14:42:13.992375 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:13.992352    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"]
Jul 17 14:42:14.057660 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.057634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b1c6c1-926f-452a-a3aa-a230564cb246-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-729mb\" (UID: \"69b1c6c1-926f-452a-a3aa-a230564cb246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"
Jul 17 14:42:14.158963 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.158942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b1c6c1-926f-452a-a3aa-a230564cb246-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-729mb\" (UID: \"69b1c6c1-926f-452a-a3aa-a230564cb246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"
Jul 17 14:42:14.159532 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.159516    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b1c6c1-926f-452a-a3aa-a230564cb246-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-729mb\" (UID: \"69b1c6c1-926f-452a-a3aa-a230564cb246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"
Jul 17 14:42:14.294162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.294074    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"
Jul 17 14:42:14.423976 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.423936    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"]
Jul 17 14:42:14.426511 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:42:14.426479    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b1c6c1_926f_452a_a3aa_a230564cb246.slice/crio-baf9f21fa44103e906fba22da41c20b8643b46e20a9ade766eb1c284fd25f4d9 WatchSource:0}: Error finding container baf9f21fa44103e906fba22da41c20b8643b46e20a9ade766eb1c284fd25f4d9: Status 404 returned error can't find the container with id baf9f21fa44103e906fba22da41c20b8643b46e20a9ade766eb1c284fd25f4d9
Jul 17 14:42:14.778288 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.778248    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" event={"ID":"69b1c6c1-926f-452a-a3aa-a230564cb246","Type":"ContainerStarted","Data":"54ad452f85bd5db616d1a11a6c81b45e77ac7937130b68a536626b1385f6ff4d"}
Jul 17 14:42:14.778397 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.778325    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" event={"ID":"69b1c6c1-926f-452a-a3aa-a230564cb246","Type":"ContainerStarted","Data":"baf9f21fa44103e906fba22da41c20b8643b46e20a9ade766eb1c284fd25f4d9"}
Jul 17 14:42:14.796681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:14.796641    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" podStartSLOduration=1.796620462 podStartE2EDuration="1.796620462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:42:14.794940057 +0000 UTC m=+3413.182287496" watchObservedRunningTime="2026-07-17 14:42:14.796620462 +0000 UTC m=+3413.183967901"
Jul 17 14:42:15.784749 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:15.784716    2575 generic.go:356] "Generic (PLEG): container finished" podID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerID="54ad452f85bd5db616d1a11a6c81b45e77ac7937130b68a536626b1385f6ff4d" exitCode=6
Jul 17 14:42:15.785215 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:15.784814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" event={"ID":"69b1c6c1-926f-452a-a3aa-a230564cb246","Type":"ContainerDied","Data":"54ad452f85bd5db616d1a11a6c81b45e77ac7937130b68a536626b1385f6ff4d"}
Jul 17 14:42:15.785215 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:15.784939    2575 scope.go:117] "RemoveContainer" containerID="54ad452f85bd5db616d1a11a6c81b45e77ac7937130b68a536626b1385f6ff4d"
Jul 17 14:42:16.791841 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:16.791813    2575 generic.go:356] "Generic (PLEG): container finished" podID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerID="d57c108578840492c8a3c87dc18f3eb600c47ba723526c9322e69e977d4565ca" exitCode=6
Jul 17 14:42:16.792203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:16.791848    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" event={"ID":"69b1c6c1-926f-452a-a3aa-a230564cb246","Type":"ContainerDied","Data":"d57c108578840492c8a3c87dc18f3eb600c47ba723526c9322e69e977d4565ca"}
Jul 17 14:42:16.792203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:16.791894    2575 scope.go:117] "RemoveContainer" containerID="54ad452f85bd5db616d1a11a6c81b45e77ac7937130b68a536626b1385f6ff4d"
Jul 17 14:42:16.792203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:16.792083    2575 scope.go:117] "RemoveContainer" containerID="d57c108578840492c8a3c87dc18f3eb600c47ba723526c9322e69e977d4565ca"
Jul 17 14:42:16.792392 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:16.792292    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-729mb_odh-ai-gateway-infra(69b1c6c1-926f-452a-a3aa-a230564cb246)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246"
Jul 17 14:42:17.798222 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:17.798200    2575 scope.go:117] "RemoveContainer" containerID="d57c108578840492c8a3c87dc18f3eb600c47ba723526c9322e69e977d4565ca"
Jul 17 14:42:17.798590 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:17.798409    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-729mb_odh-ai-gateway-infra(69b1c6c1-926f-452a-a3aa-a230564cb246)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246"
Jul 17 14:42:19.068985 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:19.068957    2575 scope.go:117] "RemoveContainer" containerID="14fecc6bac9aa16e3be3e6b59e9cd6937f0cfed9c69c80e4115a5a127af8d210"
Jul 17 14:42:19.809817 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:19.809788    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8948d36-48a9-4844-9580-b3734ec2c235" containerID="cfa9989cb8459598886e3c8e3d3b6f9a14f189449cc016b8fe0a34ce15b38015" exitCode=6
Jul 17 14:42:19.809952 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:19.809862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" event={"ID":"e8948d36-48a9-4844-9580-b3734ec2c235","Type":"ContainerDied","Data":"cfa9989cb8459598886e3c8e3d3b6f9a14f189449cc016b8fe0a34ce15b38015"}
Jul 17 14:42:19.809952 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:19.809926    2575 scope.go:117] "RemoveContainer" containerID="14fecc6bac9aa16e3be3e6b59e9cd6937f0cfed9c69c80e4115a5a127af8d210"
Jul 17 14:42:19.810227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:19.810201    2575 scope.go:117] "RemoveContainer" containerID="cfa9989cb8459598886e3c8e3d3b6f9a14f189449cc016b8fe0a34ce15b38015"
Jul 17 14:42:19.810451 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:19.810429    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-d2gnx_odh-ai-gateway-infra(e8948d36-48a9-4844-9580-b3734ec2c235)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" podUID="e8948d36-48a9-4844-9580-b3734ec2c235"
Jul 17 14:42:20.835458 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:20.835428    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"]
Jul 17 14:42:20.963477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:20.963456    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"
Jul 17 14:42:21.013535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.013509    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8948d36-48a9-4844-9580-b3734ec2c235-maas-api-service-ca\") pod \"e8948d36-48a9-4844-9580-b3734ec2c235\" (UID: \"e8948d36-48a9-4844-9580-b3734ec2c235\") "
Jul 17 14:42:21.013825 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.013806    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8948d36-48a9-4844-9580-b3734ec2c235-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8948d36-48a9-4844-9580-b3734ec2c235" (UID: "e8948d36-48a9-4844-9580-b3734ec2c235"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:42:21.114606 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.114549    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8948d36-48a9-4844-9580-b3734ec2c235-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:42:21.822849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.822819    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"
Jul 17 14:42:21.823007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.822869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx" event={"ID":"e8948d36-48a9-4844-9580-b3734ec2c235","Type":"ContainerDied","Data":"0e9ed9089d6ae188e15aa76650bd41a9dddf268643be641ce05fdbe0b5e5a70e"}
Jul 17 14:42:21.823007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.822925    2575 scope.go:117] "RemoveContainer" containerID="cfa9989cb8459598886e3c8e3d3b6f9a14f189449cc016b8fe0a34ce15b38015"
Jul 17 14:42:21.846350 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.846321    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"]
Jul 17 14:42:21.848494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:21.848475    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d2gnx"]
Jul 17 14:42:22.075678 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:22.075616    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" path="/var/lib/kubelet/pods/e8948d36-48a9-4844-9580-b3734ec2c235/volumes"
Jul 17 14:42:22.705969 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:22.705940    2575 scope.go:117] "RemoveContainer" containerID="6850fd31bc5e87f75046a7ccc42bd260b3db720d37641d552ef33a2b625031e5"
Jul 17 14:42:22.718476 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:22.718456    2575 scope.go:117] "RemoveContainer" containerID="b2e72c90b3dc2fb4a16b2225bc531ff247771786df053ef49407b1a19c9920f7"
Jul 17 14:42:29.926244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926211    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"]
Jul 17 14:42:29.926622 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926601    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.926622 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926613    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.926622 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926623    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.926724 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926629    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.926724 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926641    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.926724 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926646    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.926724 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926713    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.926724 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.926721    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:29.931655 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.931637    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"
Jul 17 14:42:29.940227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.940194    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"]
Jul 17 14:42:29.982546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:29.982511    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c8eeb8-6c9c-4228-80d6-eb66b75548ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8\" (UID: \"85c8eeb8-6c9c-4228-80d6-eb66b75548ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"
Jul 17 14:42:30.083559 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:30.083532    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c8eeb8-6c9c-4228-80d6-eb66b75548ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8\" (UID: \"85c8eeb8-6c9c-4228-80d6-eb66b75548ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"
Jul 17 14:42:30.084067 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:30.084050    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c8eeb8-6c9c-4228-80d6-eb66b75548ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8\" (UID: \"85c8eeb8-6c9c-4228-80d6-eb66b75548ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"
Jul 17 14:42:30.243693 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:30.243671    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"
Jul 17 14:42:30.374454 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:30.374415    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"]
Jul 17 14:42:30.375207 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:42:30.375177    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c8eeb8_6c9c_4228_80d6_eb66b75548ce.slice/crio-e4db7b4566211a156aec0d20bc581a7baf7cfc128280a884774833c3f38dfc63 WatchSource:0}: Error finding container e4db7b4566211a156aec0d20bc581a7baf7cfc128280a884774833c3f38dfc63: Status 404 returned error can't find the container with id e4db7b4566211a156aec0d20bc581a7baf7cfc128280a884774833c3f38dfc63
Jul 17 14:42:30.866132 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:30.866057    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" event={"ID":"85c8eeb8-6c9c-4228-80d6-eb66b75548ce","Type":"ContainerStarted","Data":"dbc67669f7bf6f9009c848d85bc21fdf21b1d5de7d4da3de523e96ca3cc8a602"}
Jul 17 14:42:30.866132 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:30.866110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" event={"ID":"85c8eeb8-6c9c-4228-80d6-eb66b75548ce","Type":"ContainerStarted","Data":"e4db7b4566211a156aec0d20bc581a7baf7cfc128280a884774833c3f38dfc63"}
Jul 17 14:42:30.885648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:30.885596    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" podStartSLOduration=1.8855822949999999 podStartE2EDuration="1.885582295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:42:30.883975738 +0000 UTC m=+3429.271323174" watchObservedRunningTime="2026-07-17 14:42:30.885582295 +0000 UTC m=+3429.272929734"
Jul 17 14:42:31.872076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:31.872040    2575 generic.go:356] "Generic (PLEG): container finished" podID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerID="dbc67669f7bf6f9009c848d85bc21fdf21b1d5de7d4da3de523e96ca3cc8a602" exitCode=6
Jul 17 14:42:31.872468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:31.872121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" event={"ID":"85c8eeb8-6c9c-4228-80d6-eb66b75548ce","Type":"ContainerDied","Data":"dbc67669f7bf6f9009c848d85bc21fdf21b1d5de7d4da3de523e96ca3cc8a602"}
Jul 17 14:42:31.872468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:31.872257    2575 scope.go:117] "RemoveContainer" containerID="dbc67669f7bf6f9009c848d85bc21fdf21b1d5de7d4da3de523e96ca3cc8a602"
Jul 17 14:42:32.073210 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.073186    2575 scope.go:117] "RemoveContainer" containerID="d57c108578840492c8a3c87dc18f3eb600c47ba723526c9322e69e977d4565ca"
Jul 17 14:42:32.878746 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.878713    2575 generic.go:356] "Generic (PLEG): container finished" podID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerID="120f4865d3dffd342c60d0cbe554113b4b15aff721c0f804b0cd4319220d4c0a" exitCode=6
Jul 17 14:42:32.879124 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.878795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" event={"ID":"85c8eeb8-6c9c-4228-80d6-eb66b75548ce","Type":"ContainerDied","Data":"120f4865d3dffd342c60d0cbe554113b4b15aff721c0f804b0cd4319220d4c0a"}
Jul 17 14:42:32.879124 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.878853    2575 scope.go:117] "RemoveContainer" containerID="dbc67669f7bf6f9009c848d85bc21fdf21b1d5de7d4da3de523e96ca3cc8a602"
Jul 17 14:42:32.879124 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.879056    2575 scope.go:117] "RemoveContainer" containerID="120f4865d3dffd342c60d0cbe554113b4b15aff721c0f804b0cd4319220d4c0a"
Jul 17 14:42:32.879330 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:32.879251    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8_odh-ai-gateway-infra(85c8eeb8-6c9c-4228-80d6-eb66b75548ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce"
Jul 17 14:42:32.880782 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.880765    2575 generic.go:356] "Generic (PLEG): container finished" podID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerID="792e734064c4244c71bb5f5974a569d366d0c03764883c340cb38910d7bd9be2" exitCode=6
Jul 17 14:42:32.880872 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.880837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" event={"ID":"69b1c6c1-926f-452a-a3aa-a230564cb246","Type":"ContainerDied","Data":"792e734064c4244c71bb5f5974a569d366d0c03764883c340cb38910d7bd9be2"}
Jul 17 14:42:32.881089 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.881077    2575 scope.go:117] "RemoveContainer" containerID="792e734064c4244c71bb5f5974a569d366d0c03764883c340cb38910d7bd9be2"
Jul 17 14:42:32.881249 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:32.881234    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-729mb_odh-ai-gateway-infra(69b1c6c1-926f-452a-a3aa-a230564cb246)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246"
Jul 17 14:42:32.889351 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:32.889332    2575 scope.go:117] "RemoveContainer" containerID="d57c108578840492c8a3c87dc18f3eb600c47ba723526c9322e69e977d4565ca"
Jul 17 14:42:33.104132 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.104102    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"]
Jul 17 14:42:33.104584 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.104571    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8948d36-48a9-4844-9580-b3734ec2c235" containerName="revoke-keys"
Jul 17 14:42:33.109055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.109038    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"
Jul 17 14:42:33.114790 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.114767    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"]
Jul 17 14:42:33.207965 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.207914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f5acd7-3275-432d-935d-16dd7eb6537a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c\" (UID: \"b5f5acd7-3275-432d-935d-16dd7eb6537a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"
Jul 17 14:42:33.309076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.309057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f5acd7-3275-432d-935d-16dd7eb6537a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c\" (UID: \"b5f5acd7-3275-432d-935d-16dd7eb6537a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"
Jul 17 14:42:33.309625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.309607    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f5acd7-3275-432d-935d-16dd7eb6537a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c\" (UID: \"b5f5acd7-3275-432d-935d-16dd7eb6537a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"
Jul 17 14:42:33.421820 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.421800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"
Jul 17 14:42:33.753824 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.753798    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"]
Jul 17 14:42:33.755433 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:42:33.755407    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f5acd7_3275_432d_935d_16dd7eb6537a.slice/crio-ce3fd8224271bf109bf02bd8b9bed4d045b2e5284be11241305d77cf6f929541 WatchSource:0}: Error finding container ce3fd8224271bf109bf02bd8b9bed4d045b2e5284be11241305d77cf6f929541: Status 404 returned error can't find the container with id ce3fd8224271bf109bf02bd8b9bed4d045b2e5284be11241305d77cf6f929541
Jul 17 14:42:33.887906 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.887885    2575 scope.go:117] "RemoveContainer" containerID="120f4865d3dffd342c60d0cbe554113b4b15aff721c0f804b0cd4319220d4c0a"
Jul 17 14:42:33.888229 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:33.888113    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8_odh-ai-gateway-infra(85c8eeb8-6c9c-4228-80d6-eb66b75548ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce"
Jul 17 14:42:33.889418 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.889394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" event={"ID":"b5f5acd7-3275-432d-935d-16dd7eb6537a","Type":"ContainerStarted","Data":"ce3fd8224271bf109bf02bd8b9bed4d045b2e5284be11241305d77cf6f929541"}
Jul 17 14:42:33.919525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:33.919504    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"]
Jul 17 14:42:34.054672 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.054647    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"
Jul 17 14:42:34.125071 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.125043    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b1c6c1-926f-452a-a3aa-a230564cb246-maas-api-service-ca\") pod \"69b1c6c1-926f-452a-a3aa-a230564cb246\" (UID: \"69b1c6c1-926f-452a-a3aa-a230564cb246\") "
Jul 17 14:42:34.125352 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.125333    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b1c6c1-926f-452a-a3aa-a230564cb246-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69b1c6c1-926f-452a-a3aa-a230564cb246" (UID: "69b1c6c1-926f-452a-a3aa-a230564cb246"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:42:34.225995 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.225958    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b1c6c1-926f-452a-a3aa-a230564cb246-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:42:34.898108 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.898070    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerID="629de9749e0fe40e59dfc383b92fc81585c0faf11941782669953a2d8b6441f1" exitCode=6
Jul 17 14:42:34.898479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.898154    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" event={"ID":"b5f5acd7-3275-432d-935d-16dd7eb6537a","Type":"ContainerDied","Data":"629de9749e0fe40e59dfc383b92fc81585c0faf11941782669953a2d8b6441f1"}
Jul 17 14:42:34.898479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.898305    2575 scope.go:117] "RemoveContainer" containerID="629de9749e0fe40e59dfc383b92fc81585c0faf11941782669953a2d8b6441f1"
Jul 17 14:42:34.899644 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.899628    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"
Jul 17 14:42:34.899759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.899643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb" event={"ID":"69b1c6c1-926f-452a-a3aa-a230564cb246","Type":"ContainerDied","Data":"baf9f21fa44103e906fba22da41c20b8643b46e20a9ade766eb1c284fd25f4d9"}
Jul 17 14:42:34.899759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.899688    2575 scope.go:117] "RemoveContainer" containerID="792e734064c4244c71bb5f5974a569d366d0c03764883c340cb38910d7bd9be2"
Jul 17 14:42:34.932640 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.932612    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"]
Jul 17 14:42:34.941605 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:34.941584    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-729mb"]
Jul 17 14:42:35.906189 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:35.906156    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerID="567da140bee04617a840590e321991222dc3ed1050ba70e8410ab547a65657c7" exitCode=6
Jul 17 14:42:35.906625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:35.906222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" event={"ID":"b5f5acd7-3275-432d-935d-16dd7eb6537a","Type":"ContainerDied","Data":"567da140bee04617a840590e321991222dc3ed1050ba70e8410ab547a65657c7"}
Jul 17 14:42:35.906625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:35.906343    2575 scope.go:117] "RemoveContainer" containerID="629de9749e0fe40e59dfc383b92fc81585c0faf11941782669953a2d8b6441f1"
Jul 17 14:42:35.906625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:35.906509    2575 scope.go:117] "RemoveContainer" containerID="567da140bee04617a840590e321991222dc3ed1050ba70e8410ab547a65657c7"
Jul 17 14:42:35.906800 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:35.906681    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c_odh-ai-gateway-infra(b5f5acd7-3275-432d-935d-16dd7eb6537a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a"
Jul 17 14:42:36.074846 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:36.074821    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" path="/var/lib/kubelet/pods/69b1c6c1-926f-452a-a3aa-a230564cb246/volumes"
Jul 17 14:42:36.913373 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:36.913348    2575 scope.go:117] "RemoveContainer" containerID="567da140bee04617a840590e321991222dc3ed1050ba70e8410ab547a65657c7"
Jul 17 14:42:36.913738 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:36.913527    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c_odh-ai-gateway-infra(b5f5acd7-3275-432d-935d-16dd7eb6537a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a"
Jul 17 14:42:47.069371 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:47.069344    2575 scope.go:117] "RemoveContainer" containerID="120f4865d3dffd342c60d0cbe554113b4b15aff721c0f804b0cd4319220d4c0a"
Jul 17 14:42:47.968653 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:47.968615    2575 generic.go:356] "Generic (PLEG): container finished" podID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerID="4fade59b439a6c344dd07479d77fde77ac1467c151c45e5bf0902ccc52ada92d" exitCode=6
Jul 17 14:42:47.968832 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:47.968695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" event={"ID":"85c8eeb8-6c9c-4228-80d6-eb66b75548ce","Type":"ContainerDied","Data":"4fade59b439a6c344dd07479d77fde77ac1467c151c45e5bf0902ccc52ada92d"}
Jul 17 14:42:47.968832 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:47.968756    2575 scope.go:117] "RemoveContainer" containerID="120f4865d3dffd342c60d0cbe554113b4b15aff721c0f804b0cd4319220d4c0a"
Jul 17 14:42:47.969041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:47.969017    2575 scope.go:117] "RemoveContainer" containerID="4fade59b439a6c344dd07479d77fde77ac1467c151c45e5bf0902ccc52ada92d"
Jul 17 14:42:47.969215 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:47.969194    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8_odh-ai-gateway-infra(85c8eeb8-6c9c-4228-80d6-eb66b75548ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce"
Jul 17 14:42:48.999989 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:48.999961    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"]
Jul 17 14:42:49.069032 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.069008    2575 scope.go:117] "RemoveContainer" containerID="567da140bee04617a840590e321991222dc3ed1050ba70e8410ab547a65657c7"
Jul 17 14:42:49.138071 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.138048    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"
Jul 17 14:42:49.253593 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.253523    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c8eeb8-6c9c-4228-80d6-eb66b75548ce-maas-api-service-ca\") pod \"85c8eeb8-6c9c-4228-80d6-eb66b75548ce\" (UID: \"85c8eeb8-6c9c-4228-80d6-eb66b75548ce\") "
Jul 17 14:42:49.253886 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.253859    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c8eeb8-6c9c-4228-80d6-eb66b75548ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85c8eeb8-6c9c-4228-80d6-eb66b75548ce" (UID: "85c8eeb8-6c9c-4228-80d6-eb66b75548ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:42:49.354348 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.354323    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c8eeb8-6c9c-4228-80d6-eb66b75548ce-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:42:49.981322 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.981271    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"
Jul 17 14:42:49.981515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.981318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8" event={"ID":"85c8eeb8-6c9c-4228-80d6-eb66b75548ce","Type":"ContainerDied","Data":"e4db7b4566211a156aec0d20bc581a7baf7cfc128280a884774833c3f38dfc63"}
Jul 17 14:42:49.981515 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.981406    2575 scope.go:117] "RemoveContainer" containerID="4fade59b439a6c344dd07479d77fde77ac1467c151c45e5bf0902ccc52ada92d"
Jul 17 14:42:49.983249 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.983218    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerID="1c1194e45d929b7e6cc83c2a18a3d7dcd2d76039d098a17beeb828a4ed4c5bbd" exitCode=6
Jul 17 14:42:49.983394 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.983298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" event={"ID":"b5f5acd7-3275-432d-935d-16dd7eb6537a","Type":"ContainerDied","Data":"1c1194e45d929b7e6cc83c2a18a3d7dcd2d76039d098a17beeb828a4ed4c5bbd"}
Jul 17 14:42:49.983570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.983555    2575 scope.go:117] "RemoveContainer" containerID="1c1194e45d929b7e6cc83c2a18a3d7dcd2d76039d098a17beeb828a4ed4c5bbd"
Jul 17 14:42:49.983760 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:49.983743    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c_odh-ai-gateway-infra(b5f5acd7-3275-432d-935d-16dd7eb6537a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a"
Jul 17 14:42:49.990228 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:49.990209    2575 scope.go:117] "RemoveContainer" containerID="567da140bee04617a840590e321991222dc3ed1050ba70e8410ab547a65657c7"
Jul 17 14:42:50.018381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:50.018325    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"]
Jul 17 14:42:50.023330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:50.023264    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-g9dn8"]
Jul 17 14:42:50.074513 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:50.074487    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" path="/var/lib/kubelet/pods/85c8eeb8-6c9c-4228-80d6-eb66b75548ce/volumes"
Jul 17 14:42:51.015158 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.015129    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"]
Jul 17 14:42:51.145706 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.145676    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"
Jul 17 14:42:51.274092 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.274023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f5acd7-3275-432d-935d-16dd7eb6537a-maas-api-service-ca\") pod \"b5f5acd7-3275-432d-935d-16dd7eb6537a\" (UID: \"b5f5acd7-3275-432d-935d-16dd7eb6537a\") "
Jul 17 14:42:51.274409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.274381    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f5acd7-3275-432d-935d-16dd7eb6537a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5f5acd7-3275-432d-935d-16dd7eb6537a" (UID: "b5f5acd7-3275-432d-935d-16dd7eb6537a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:42:51.374884 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.374853    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f5acd7-3275-432d-935d-16dd7eb6537a-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:42:51.998269 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.998238    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"
Jul 17 14:42:51.998451 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.998290    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c" event={"ID":"b5f5acd7-3275-432d-935d-16dd7eb6537a","Type":"ContainerDied","Data":"ce3fd8224271bf109bf02bd8b9bed4d045b2e5284be11241305d77cf6f929541"}
Jul 17 14:42:51.998451 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:51.998349    2575 scope.go:117] "RemoveContainer" containerID="1c1194e45d929b7e6cc83c2a18a3d7dcd2d76039d098a17beeb828a4ed4c5bbd"
Jul 17 14:42:52.019710 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.019683    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"]
Jul 17 14:42:52.023342 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.023322    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-5mt6c"]
Jul 17 14:42:52.075164 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.075141    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" path="/var/lib/kubelet/pods/b5f5acd7-3275-432d-935d-16dd7eb6537a/volumes"
Jul 17 14:42:52.392984 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.392906    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"]
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393289    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393300    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393310    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393316    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393321    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393327    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393345    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393351    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393356    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:42:52.393355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393361    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393374    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393379    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393385    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393389    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393440    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393448    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393454    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393461    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393465    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:42:52.393658 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.393470    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:42:52.396603 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.396585    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"
Jul 17 14:42:52.398814 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.398795    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:42:52.403007 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.402984    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"]
Jul 17 14:42:52.484326 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.484272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c512f-fdd1-4e2e-9f9a-1211a9cd9df0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-g9chw\" (UID: \"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"
Jul 17 14:42:52.585679 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.585642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c512f-fdd1-4e2e-9f9a-1211a9cd9df0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-g9chw\" (UID: \"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"
Jul 17 14:42:52.586243 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.586223    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c512f-fdd1-4e2e-9f9a-1211a9cd9df0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-g9chw\" (UID: \"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"
Jul 17 14:42:52.707847 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.707767    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"
Jul 17 14:42:52.836472 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:52.836449    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"]
Jul 17 14:42:52.838716 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:42:52.838686    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690c512f_fdd1_4e2e_9f9a_1211a9cd9df0.slice/crio-331c56fd58c6aa677f37537554eb6b2aab6cf6785cf121b1255c44ab7e9da0cf WatchSource:0}: Error finding container 331c56fd58c6aa677f37537554eb6b2aab6cf6785cf121b1255c44ab7e9da0cf: Status 404 returned error can't find the container with id 331c56fd58c6aa677f37537554eb6b2aab6cf6785cf121b1255c44ab7e9da0cf
Jul 17 14:42:53.004796 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:53.004765    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" event={"ID":"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0","Type":"ContainerStarted","Data":"331c56fd58c6aa677f37537554eb6b2aab6cf6785cf121b1255c44ab7e9da0cf"}
Jul 17 14:42:54.011131 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:54.011099    2575 generic.go:356] "Generic (PLEG): container finished" podID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerID="dd436c09108212b23895e03bc8d18db384fb156f7103123e0a90c7ab1614cfc6" exitCode=6
Jul 17 14:42:54.011532 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:54.011187    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" event={"ID":"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0","Type":"ContainerDied","Data":"dd436c09108212b23895e03bc8d18db384fb156f7103123e0a90c7ab1614cfc6"}
Jul 17 14:42:54.011532 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:54.011358    2575 scope.go:117] "RemoveContainer" containerID="dd436c09108212b23895e03bc8d18db384fb156f7103123e0a90c7ab1614cfc6"
Jul 17 14:42:55.017883 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:55.017844    2575 generic.go:356] "Generic (PLEG): container finished" podID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerID="9c87fc4301d904f5ef10b1a3bf5ba32b35c4dd83050cd2392b228e8387d146e2" exitCode=6
Jul 17 14:42:55.018362 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:55.017923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" event={"ID":"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0","Type":"ContainerDied","Data":"9c87fc4301d904f5ef10b1a3bf5ba32b35c4dd83050cd2392b228e8387d146e2"}
Jul 17 14:42:55.018362 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:55.018035    2575 scope.go:117] "RemoveContainer" containerID="dd436c09108212b23895e03bc8d18db384fb156f7103123e0a90c7ab1614cfc6"
Jul 17 14:42:55.018362 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:55.018264    2575 scope.go:117] "RemoveContainer" containerID="9c87fc4301d904f5ef10b1a3bf5ba32b35c4dd83050cd2392b228e8387d146e2"
Jul 17 14:42:55.018536 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:55.018506    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-g9chw_odh-ai-gateway-infra(690c512f-fdd1-4e2e-9f9a-1211a9cd9df0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0"
Jul 17 14:42:56.024745 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:42:56.024719    2575 scope.go:117] "RemoveContainer" containerID="9c87fc4301d904f5ef10b1a3bf5ba32b35c4dd83050cd2392b228e8387d146e2"
Jul 17 14:42:56.025095 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:42:56.024904    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-g9chw_odh-ai-gateway-infra(690c512f-fdd1-4e2e-9f9a-1211a9cd9df0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0"
Jul 17 14:43:09.069846 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:09.069811    2575 scope.go:117] "RemoveContainer" containerID="9c87fc4301d904f5ef10b1a3bf5ba32b35c4dd83050cd2392b228e8387d146e2"
Jul 17 14:43:10.095270 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:10.095236    2575 generic.go:356] "Generic (PLEG): container finished" podID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerID="99c3230144b47ef7296bdfceeca04c3df992799185f0ba8acddbced6cb728e14" exitCode=6
Jul 17 14:43:10.095270 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:10.095269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" event={"ID":"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0","Type":"ContainerDied","Data":"99c3230144b47ef7296bdfceeca04c3df992799185f0ba8acddbced6cb728e14"}
Jul 17 14:43:10.095722 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:10.095332    2575 scope.go:117] "RemoveContainer" containerID="9c87fc4301d904f5ef10b1a3bf5ba32b35c4dd83050cd2392b228e8387d146e2"
Jul 17 14:43:10.095722 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:10.095595    2575 scope.go:117] "RemoveContainer" containerID="99c3230144b47ef7296bdfceeca04c3df992799185f0ba8acddbced6cb728e14"
Jul 17 14:43:10.095866 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:10.095832    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-g9chw_odh-ai-gateway-infra(690c512f-fdd1-4e2e-9f9a-1211a9cd9df0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0"
Jul 17 14:43:11.123648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:11.123620    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"]
Jul 17 14:43:11.254733 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:11.254707    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"
Jul 17 14:43:11.351642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:11.351608    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c512f-fdd1-4e2e-9f9a-1211a9cd9df0-maas-api-service-ca\") pod \"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0\" (UID: \"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0\") "
Jul 17 14:43:11.351933 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:11.351911    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690c512f-fdd1-4e2e-9f9a-1211a9cd9df0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" (UID: "690c512f-fdd1-4e2e-9f9a-1211a9cd9df0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:43:11.452751 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:11.452676    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c512f-fdd1-4e2e-9f9a-1211a9cd9df0-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:43:12.107523 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:12.107498    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"
Jul 17 14:43:12.107643 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:12.107497    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw" event={"ID":"690c512f-fdd1-4e2e-9f9a-1211a9cd9df0","Type":"ContainerDied","Data":"331c56fd58c6aa677f37537554eb6b2aab6cf6785cf121b1255c44ab7e9da0cf"}
Jul 17 14:43:12.107643 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:12.107632    2575 scope.go:117] "RemoveContainer" containerID="99c3230144b47ef7296bdfceeca04c3df992799185f0ba8acddbced6cb728e14"
Jul 17 14:43:12.125141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:12.125113    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"]
Jul 17 14:43:12.128958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:12.128938    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-g9chw"]
Jul 17 14:43:14.019495 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019462    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"]
Jul 17 14:43:14.019863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019837    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.019863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019847    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.019863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019855    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:43:14.019863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019861    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019877    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019883    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019893    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019898    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019905    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019910    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019967    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b1c6c1-926f-452a-a3aa-a230564cb246" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019974    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019979    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c8eeb8-6c9c-4228-80d6-eb66b75548ce" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019986    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:14.020001 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.019991    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f5acd7-3275-432d-935d-16dd7eb6537a" containerName="revoke-keys"
Jul 17 14:43:14.024330 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.024313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"
Jul 17 14:43:14.026602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.026581    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:43:14.030948 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.030925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"]
Jul 17 14:43:14.074636 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.074609    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" path="/var/lib/kubelet/pods/690c512f-fdd1-4e2e-9f9a-1211a9cd9df0/volumes"
Jul 17 14:43:14.074957 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.074939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ca73a7-925e-4751-82ca-6fb3c2ad27a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm\" (UID: \"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"
Jul 17 14:43:14.175807 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.175779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ca73a7-925e-4751-82ca-6fb3c2ad27a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm\" (UID: \"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"
Jul 17 14:43:14.176379 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.176361    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ca73a7-925e-4751-82ca-6fb3c2ad27a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm\" (UID: \"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"
Jul 17 14:43:14.336932 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.336866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"
Jul 17 14:43:14.464162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:14.464135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"]
Jul 17 14:43:14.466317 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:43:14.466288    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ca73a7_925e_4751_82ca_6fb3c2ad27a2.slice/crio-e08fa413f76ad732643801af90316e5c15d11f7115a11d21a06f28801858882d WatchSource:0}: Error finding container e08fa413f76ad732643801af90316e5c15d11f7115a11d21a06f28801858882d: Status 404 returned error can't find the container with id e08fa413f76ad732643801af90316e5c15d11f7115a11d21a06f28801858882d
Jul 17 14:43:15.125646 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:15.125612    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerID="265099409f1e4b35131557681c3ea5f3c19ad1524ed854d6a1d643f4ba9441eb" exitCode=6
Jul 17 14:43:15.125997 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:15.125660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" event={"ID":"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2","Type":"ContainerDied","Data":"265099409f1e4b35131557681c3ea5f3c19ad1524ed854d6a1d643f4ba9441eb"}
Jul 17 14:43:15.125997 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:15.125698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" event={"ID":"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2","Type":"ContainerStarted","Data":"e08fa413f76ad732643801af90316e5c15d11f7115a11d21a06f28801858882d"}
Jul 17 14:43:15.125997 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:15.125874    2575 scope.go:117] "RemoveContainer" containerID="265099409f1e4b35131557681c3ea5f3c19ad1524ed854d6a1d643f4ba9441eb"
Jul 17 14:43:16.133652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:16.133572    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerID="4f39598e18f399f806f5b5fd2da79554831afc2c3700e91742f4d00160e2d207" exitCode=6
Jul 17 14:43:16.134045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:16.133651    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" event={"ID":"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2","Type":"ContainerDied","Data":"4f39598e18f399f806f5b5fd2da79554831afc2c3700e91742f4d00160e2d207"}
Jul 17 14:43:16.134045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:16.133700    2575 scope.go:117] "RemoveContainer" containerID="265099409f1e4b35131557681c3ea5f3c19ad1524ed854d6a1d643f4ba9441eb"
Jul 17 14:43:16.134045 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:16.133910    2575 scope.go:117] "RemoveContainer" containerID="4f39598e18f399f806f5b5fd2da79554831afc2c3700e91742f4d00160e2d207"
Jul 17 14:43:16.134167 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:16.134102    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm_odh-ai-gateway-infra(d2ca73a7-925e-4751-82ca-6fb3c2ad27a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2"
Jul 17 14:43:17.140977 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:17.140947    2575 scope.go:117] "RemoveContainer" containerID="4f39598e18f399f806f5b5fd2da79554831afc2c3700e91742f4d00160e2d207"
Jul 17 14:43:17.141373 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:17.141146    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm_odh-ai-gateway-infra(d2ca73a7-925e-4751-82ca-6fb3c2ad27a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2"
Jul 17 14:43:22.759957 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:22.759930    2575 scope.go:117] "RemoveContainer" containerID="c0006ae37c2e2862ad020430b54f4dedbdee2a4f570dd19ce3a4957936c65fea"
Jul 17 14:43:22.767916 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:22.767898    2575 scope.go:117] "RemoveContainer" containerID="b5525b96243f805ede343dfe06191873eb07ead69a64887375d727988f7db919"
Jul 17 14:43:23.174479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.174411    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"]
Jul 17 14:43:23.174874 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.174861    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="690c512f-fdd1-4e2e-9f9a-1211a9cd9df0" containerName="revoke-keys"
Jul 17 14:43:23.179382 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.179353    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"
Jul 17 14:43:23.195807 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.195784    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"]
Jul 17 14:43:23.247626 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.247600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8cf2d6a-9a68-4615-b75d-66554bb3a86d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq\" (UID: \"d8cf2d6a-9a68-4615-b75d-66554bb3a86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"
Jul 17 14:43:23.348162 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.348132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8cf2d6a-9a68-4615-b75d-66554bb3a86d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq\" (UID: \"d8cf2d6a-9a68-4615-b75d-66554bb3a86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"
Jul 17 14:43:23.348752 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.348733    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8cf2d6a-9a68-4615-b75d-66554bb3a86d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq\" (UID: \"d8cf2d6a-9a68-4615-b75d-66554bb3a86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"
Jul 17 14:43:23.489749 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.489726    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"
Jul 17 14:43:23.623788 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:23.623642    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"]
Jul 17 14:43:23.626484 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:43:23.626455    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cf2d6a_9a68_4615_b75d_66554bb3a86d.slice/crio-597b390660bb11ca0836a07a08a7894df875a08a407e11005d310fb5a1aa95ce WatchSource:0}: Error finding container 597b390660bb11ca0836a07a08a7894df875a08a407e11005d310fb5a1aa95ce: Status 404 returned error can't find the container with id 597b390660bb11ca0836a07a08a7894df875a08a407e11005d310fb5a1aa95ce
Jul 17 14:43:24.175219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:24.175134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" event={"ID":"d8cf2d6a-9a68-4615-b75d-66554bb3a86d","Type":"ContainerStarted","Data":"3165bca443d7d2799fd5555cdc2dfb313606104b5d4884e93bd4cc9d6b1a447b"}
Jul 17 14:43:24.175219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:24.175192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" event={"ID":"d8cf2d6a-9a68-4615-b75d-66554bb3a86d","Type":"ContainerStarted","Data":"597b390660bb11ca0836a07a08a7894df875a08a407e11005d310fb5a1aa95ce"}
Jul 17 14:43:24.196513 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:24.196469    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" podStartSLOduration=1.19645536 podStartE2EDuration="1.19645536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:43:24.195026809 +0000 UTC m=+3482.582374250" watchObservedRunningTime="2026-07-17 14:43:24.19645536 +0000 UTC m=+3482.583802799"
Jul 17 14:43:25.182381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:25.182346    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerID="3165bca443d7d2799fd5555cdc2dfb313606104b5d4884e93bd4cc9d6b1a447b" exitCode=6
Jul 17 14:43:25.182798 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:25.182413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" event={"ID":"d8cf2d6a-9a68-4615-b75d-66554bb3a86d","Type":"ContainerDied","Data":"3165bca443d7d2799fd5555cdc2dfb313606104b5d4884e93bd4cc9d6b1a447b"}
Jul 17 14:43:25.182798 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:25.182603    2575 scope.go:117] "RemoveContainer" containerID="3165bca443d7d2799fd5555cdc2dfb313606104b5d4884e93bd4cc9d6b1a447b"
Jul 17 14:43:26.188309 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:26.188262    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerID="a11275e2e12852460bbabfb6be89540bae8888baef074f34bb75637ae3a40340" exitCode=6
Jul 17 14:43:26.188626 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:26.188348    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" event={"ID":"d8cf2d6a-9a68-4615-b75d-66554bb3a86d","Type":"ContainerDied","Data":"a11275e2e12852460bbabfb6be89540bae8888baef074f34bb75637ae3a40340"}
Jul 17 14:43:26.188626 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:26.188411    2575 scope.go:117] "RemoveContainer" containerID="3165bca443d7d2799fd5555cdc2dfb313606104b5d4884e93bd4cc9d6b1a447b"
Jul 17 14:43:26.188626 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:26.188605    2575 scope.go:117] "RemoveContainer" containerID="a11275e2e12852460bbabfb6be89540bae8888baef074f34bb75637ae3a40340"
Jul 17 14:43:26.188795 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:26.188783    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq_odh-ai-gateway-infra(d8cf2d6a-9a68-4615-b75d-66554bb3a86d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d"
Jul 17 14:43:27.195076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:27.195052    2575 scope.go:117] "RemoveContainer" containerID="a11275e2e12852460bbabfb6be89540bae8888baef074f34bb75637ae3a40340"
Jul 17 14:43:27.195450 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:27.195204    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq_odh-ai-gateway-infra(d8cf2d6a-9a68-4615-b75d-66554bb3a86d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d"
Jul 17 14:43:29.999523 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:29.999492    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"]
Jul 17 14:43:30.004546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:30.004525    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"
Jul 17 14:43:30.015355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:30.015334    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"]
Jul 17 14:43:30.105962 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:30.105936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d564f-9b37-4819-9d6d-e9022a2537fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl\" (UID: \"cc0d564f-9b37-4819-9d6d-e9022a2537fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"
Jul 17 14:43:30.206592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:30.206567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d564f-9b37-4819-9d6d-e9022a2537fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl\" (UID: \"cc0d564f-9b37-4819-9d6d-e9022a2537fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"
Jul 17 14:43:30.207201 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:30.207180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d564f-9b37-4819-9d6d-e9022a2537fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl\" (UID: \"cc0d564f-9b37-4819-9d6d-e9022a2537fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"
Jul 17 14:43:30.315168 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:30.315111    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"
Jul 17 14:43:30.453106 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:30.453077    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"]
Jul 17 14:43:30.454714 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:43:30.454691    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0d564f_9b37_4819_9d6d_e9022a2537fd.slice/crio-76efcd12139b36fd7e4b7c2a68c6fffd3d5dfba3cb21547ddecb2d587932456b WatchSource:0}: Error finding container 76efcd12139b36fd7e4b7c2a68c6fffd3d5dfba3cb21547ddecb2d587932456b: Status 404 returned error can't find the container with id 76efcd12139b36fd7e4b7c2a68c6fffd3d5dfba3cb21547ddecb2d587932456b
Jul 17 14:43:31.215937 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:31.215905    2575 generic.go:356] "Generic (PLEG): container finished" podID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerID="39806ed9b4d8c7d11f645a2e694baf5ebb9bf3efb4ecb6628e12442d3de48251" exitCode=6
Jul 17 14:43:31.216244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:31.215970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" event={"ID":"cc0d564f-9b37-4819-9d6d-e9022a2537fd","Type":"ContainerDied","Data":"39806ed9b4d8c7d11f645a2e694baf5ebb9bf3efb4ecb6628e12442d3de48251"}
Jul 17 14:43:31.216244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:31.216011    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" event={"ID":"cc0d564f-9b37-4819-9d6d-e9022a2537fd","Type":"ContainerStarted","Data":"76efcd12139b36fd7e4b7c2a68c6fffd3d5dfba3cb21547ddecb2d587932456b"}
Jul 17 14:43:31.216244 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:31.216134    2575 scope.go:117] "RemoveContainer" containerID="39806ed9b4d8c7d11f645a2e694baf5ebb9bf3efb4ecb6628e12442d3de48251"
Jul 17 14:43:32.071712 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:32.071635    2575 scope.go:117] "RemoveContainer" containerID="4f39598e18f399f806f5b5fd2da79554831afc2c3700e91742f4d00160e2d207"
Jul 17 14:43:32.223095 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:32.223064    2575 generic.go:356] "Generic (PLEG): container finished" podID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerID="7d3443a69ccbf2b041af8d8cc271cb4fd31b227d3dfdf1a61634a0f4cc81795c" exitCode=6
Jul 17 14:43:32.223468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:32.223110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" event={"ID":"cc0d564f-9b37-4819-9d6d-e9022a2537fd","Type":"ContainerDied","Data":"7d3443a69ccbf2b041af8d8cc271cb4fd31b227d3dfdf1a61634a0f4cc81795c"}
Jul 17 14:43:32.223468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:32.223173    2575 scope.go:117] "RemoveContainer" containerID="39806ed9b4d8c7d11f645a2e694baf5ebb9bf3efb4ecb6628e12442d3de48251"
Jul 17 14:43:32.223468 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:32.223329    2575 scope.go:117] "RemoveContainer" containerID="7d3443a69ccbf2b041af8d8cc271cb4fd31b227d3dfdf1a61634a0f4cc81795c"
Jul 17 14:43:32.223634 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:32.223516    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl_odh-ai-gateway-infra(cc0d564f-9b37-4819-9d6d-e9022a2537fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd"
Jul 17 14:43:33.229895 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:33.229862    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerID="ec46e6aeacfd471eab0760473a0d19c32c0c41f941751a24d317e7333abc6b37" exitCode=6
Jul 17 14:43:33.230381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:33.229941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" event={"ID":"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2","Type":"ContainerDied","Data":"ec46e6aeacfd471eab0760473a0d19c32c0c41f941751a24d317e7333abc6b37"}
Jul 17 14:43:33.230381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:33.230014    2575 scope.go:117] "RemoveContainer" containerID="4f39598e18f399f806f5b5fd2da79554831afc2c3700e91742f4d00160e2d207"
Jul 17 14:43:33.230381 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:33.230346    2575 scope.go:117] "RemoveContainer" containerID="ec46e6aeacfd471eab0760473a0d19c32c0c41f941751a24d317e7333abc6b37"
Jul 17 14:43:33.230598 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:33.230578    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm_odh-ai-gateway-infra(d2ca73a7-925e-4751-82ca-6fb3c2ad27a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2"
Jul 17 14:43:33.231907 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:33.231890    2575 scope.go:117] "RemoveContainer" containerID="7d3443a69ccbf2b041af8d8cc271cb4fd31b227d3dfdf1a61634a0f4cc81795c"
Jul 17 14:43:33.232038 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:33.232023    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl_odh-ai-gateway-infra(cc0d564f-9b37-4819-9d6d-e9022a2537fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd"
Jul 17 14:43:34.261288 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:34.261241    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"]
Jul 17 14:43:34.397525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:34.397503    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"
Jul 17 14:43:34.542544 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:34.542455    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ca73a7-925e-4751-82ca-6fb3c2ad27a2-maas-api-service-ca\") pod \"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2\" (UID: \"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2\") "
Jul 17 14:43:34.542809 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:34.542786    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ca73a7-925e-4751-82ca-6fb3c2ad27a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" (UID: "d2ca73a7-925e-4751-82ca-6fb3c2ad27a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:43:34.647959 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:34.644251    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ca73a7-925e-4751-82ca-6fb3c2ad27a2-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:43:35.244366 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:35.244337    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"
Jul 17 14:43:35.244542 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:35.244372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm" event={"ID":"d2ca73a7-925e-4751-82ca-6fb3c2ad27a2","Type":"ContainerDied","Data":"e08fa413f76ad732643801af90316e5c15d11f7115a11d21a06f28801858882d"}
Jul 17 14:43:35.244542 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:35.244433    2575 scope.go:117] "RemoveContainer" containerID="ec46e6aeacfd471eab0760473a0d19c32c0c41f941751a24d317e7333abc6b37"
Jul 17 14:43:35.267401 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:35.267357    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"]
Jul 17 14:43:35.269457 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:35.269439    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-b8ntm"]
Jul 17 14:43:36.074373 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:36.074340    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" path="/var/lib/kubelet/pods/d2ca73a7-925e-4751-82ca-6fb3c2ad27a2/volumes"
Jul 17 14:43:39.069296 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:39.069243    2575 scope.go:117] "RemoveContainer" containerID="a11275e2e12852460bbabfb6be89540bae8888baef074f34bb75637ae3a40340"
Jul 17 14:43:40.271511 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:40.271480    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerID="6a4e0d7828f94ae93ed50df6e0476184cb36c0da0b22fff1a7d8ef83f1e1cffc" exitCode=6
Jul 17 14:43:40.271921 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:40.271517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" event={"ID":"d8cf2d6a-9a68-4615-b75d-66554bb3a86d","Type":"ContainerDied","Data":"6a4e0d7828f94ae93ed50df6e0476184cb36c0da0b22fff1a7d8ef83f1e1cffc"}
Jul 17 14:43:40.271921 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:40.271564    2575 scope.go:117] "RemoveContainer" containerID="a11275e2e12852460bbabfb6be89540bae8888baef074f34bb75637ae3a40340"
Jul 17 14:43:40.271921 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:40.271846    2575 scope.go:117] "RemoveContainer" containerID="6a4e0d7828f94ae93ed50df6e0476184cb36c0da0b22fff1a7d8ef83f1e1cffc"
Jul 17 14:43:40.272081 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:40.272062    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq_odh-ai-gateway-infra(d8cf2d6a-9a68-4615-b75d-66554bb3a86d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d"
Jul 17 14:43:41.299362 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:41.299330    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"]
Jul 17 14:43:41.435239 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:41.435212    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"
Jul 17 14:43:41.508211 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:41.508184    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8cf2d6a-9a68-4615-b75d-66554bb3a86d-maas-api-service-ca\") pod \"d8cf2d6a-9a68-4615-b75d-66554bb3a86d\" (UID: \"d8cf2d6a-9a68-4615-b75d-66554bb3a86d\") "
Jul 17 14:43:41.508574 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:41.508549    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8cf2d6a-9a68-4615-b75d-66554bb3a86d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8cf2d6a-9a68-4615-b75d-66554bb3a86d" (UID: "d8cf2d6a-9a68-4615-b75d-66554bb3a86d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:43:41.609494 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:41.609420    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8cf2d6a-9a68-4615-b75d-66554bb3a86d-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:43:42.284480 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.284451    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"
Jul 17 14:43:42.284659 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.284482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq" event={"ID":"d8cf2d6a-9a68-4615-b75d-66554bb3a86d","Type":"ContainerDied","Data":"597b390660bb11ca0836a07a08a7894df875a08a407e11005d310fb5a1aa95ce"}
Jul 17 14:43:42.284659 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.284542    2575 scope.go:117] "RemoveContainer" containerID="6a4e0d7828f94ae93ed50df6e0476184cb36c0da0b22fff1a7d8ef83f1e1cffc"
Jul 17 14:43:42.310011 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.309985    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"]
Jul 17 14:43:42.317012 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.316984    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-mntcq"]
Jul 17 14:43:42.431829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.431760    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"]
Jul 17 14:43:42.432181 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432168    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:43:42.432220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432183    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:43:42.432220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432202    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.432220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432208    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.432220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432213    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:43:42.432220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432219    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432227    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432235    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432255    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432260    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432336    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432343    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432349    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.432415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.432355    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:43:42.436733 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.436713    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"
Jul 17 14:43:42.443854 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.443832    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"]
Jul 17 14:43:42.516262 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.516232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-cdrll\" (UID: \"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"
Jul 17 14:43:42.617121 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.617089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-cdrll\" (UID: \"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"
Jul 17 14:43:42.617699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.617680    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-cdrll\" (UID: \"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"
Jul 17 14:43:42.749889 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:42.749862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"
Jul 17 14:43:43.090554 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:43.090530    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"]
Jul 17 14:43:43.092231 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:43:43.092203    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f406aa8_208b_4d4d_8fd9_2d7ee0b06ef0.slice/crio-9ff03fb7607dedfd035b47d82b0b8ce3086f22c92e1ae8ed51407f7d24703566 WatchSource:0}: Error finding container 9ff03fb7607dedfd035b47d82b0b8ce3086f22c92e1ae8ed51407f7d24703566: Status 404 returned error can't find the container with id 9ff03fb7607dedfd035b47d82b0b8ce3086f22c92e1ae8ed51407f7d24703566
Jul 17 14:43:43.291487 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:43.291455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" event={"ID":"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0","Type":"ContainerStarted","Data":"9ff03fb7607dedfd035b47d82b0b8ce3086f22c92e1ae8ed51407f7d24703566"}
Jul 17 14:43:44.074999 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:44.074969    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" path="/var/lib/kubelet/pods/d8cf2d6a-9a68-4615-b75d-66554bb3a86d/volumes"
Jul 17 14:43:44.298694 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:44.298622    2575 generic.go:356] "Generic (PLEG): container finished" podID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerID="2262533e371d99bfad889d96df11d605ad73c938e475ef49d6d4eac938bd2b56" exitCode=6
Jul 17 14:43:44.298840 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:44.298712    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" event={"ID":"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0","Type":"ContainerDied","Data":"2262533e371d99bfad889d96df11d605ad73c938e475ef49d6d4eac938bd2b56"}
Jul 17 14:43:44.298926 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:44.298882    2575 scope.go:117] "RemoveContainer" containerID="2262533e371d99bfad889d96df11d605ad73c938e475ef49d6d4eac938bd2b56"
Jul 17 14:43:45.306143 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:45.306106    2575 generic.go:356] "Generic (PLEG): container finished" podID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerID="1de49b64cd97f2d0cdaa3c04db83bd3d3802125b4447a65f6ec48b2fe0e1573b" exitCode=6
Jul 17 14:43:45.306612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:45.306180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" event={"ID":"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0","Type":"ContainerDied","Data":"1de49b64cd97f2d0cdaa3c04db83bd3d3802125b4447a65f6ec48b2fe0e1573b"}
Jul 17 14:43:45.306612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:45.306237    2575 scope.go:117] "RemoveContainer" containerID="2262533e371d99bfad889d96df11d605ad73c938e475ef49d6d4eac938bd2b56"
Jul 17 14:43:45.306612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:45.306435    2575 scope.go:117] "RemoveContainer" containerID="1de49b64cd97f2d0cdaa3c04db83bd3d3802125b4447a65f6ec48b2fe0e1573b"
Jul 17 14:43:45.306725 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:45.306667    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-cdrll_odh-ai-gateway-infra(4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0"
Jul 17 14:43:46.069259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:46.069226    2575 scope.go:117] "RemoveContainer" containerID="7d3443a69ccbf2b041af8d8cc271cb4fd31b227d3dfdf1a61634a0f4cc81795c"
Jul 17 14:43:46.312936 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:46.312905    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" event={"ID":"cc0d564f-9b37-4819-9d6d-e9022a2537fd","Type":"ContainerStarted","Data":"3c33af957f52b7e34362101d87e1a304c281583863e8381bad92c4f37c8463e7"}
Jul 17 14:43:46.314667 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:46.314647    2575 scope.go:117] "RemoveContainer" containerID="1de49b64cd97f2d0cdaa3c04db83bd3d3802125b4447a65f6ec48b2fe0e1573b"
Jul 17 14:43:46.314853 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:46.314836    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-cdrll_odh-ai-gateway-infra(4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0"
Jul 17 14:43:46.333031 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:46.332956    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" podStartSLOduration=17.332943653 podStartE2EDuration="17.332943653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:43:46.330418836 +0000 UTC m=+3504.717766290" watchObservedRunningTime="2026-07-17 14:43:46.332943653 +0000 UTC m=+3504.720291092"
Jul 17 14:43:47.321355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:47.321319    2575 generic.go:356] "Generic (PLEG): container finished" podID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerID="3c33af957f52b7e34362101d87e1a304c281583863e8381bad92c4f37c8463e7" exitCode=6
Jul 17 14:43:47.321813 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:47.321397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" event={"ID":"cc0d564f-9b37-4819-9d6d-e9022a2537fd","Type":"ContainerDied","Data":"3c33af957f52b7e34362101d87e1a304c281583863e8381bad92c4f37c8463e7"}
Jul 17 14:43:47.321813 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:47.321469    2575 scope.go:117] "RemoveContainer" containerID="7d3443a69ccbf2b041af8d8cc271cb4fd31b227d3dfdf1a61634a0f4cc81795c"
Jul 17 14:43:47.321938 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:47.321858    2575 scope.go:117] "RemoveContainer" containerID="3c33af957f52b7e34362101d87e1a304c281583863e8381bad92c4f37c8463e7"
Jul 17 14:43:47.322155 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:47.322132    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl_odh-ai-gateway-infra(cc0d564f-9b37-4819-9d6d-e9022a2537fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd"
Jul 17 14:43:47.351670 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:47.351647    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"]
Jul 17 14:43:48.482035 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:48.482009    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"
Jul 17 14:43:48.561602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:48.561557    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d564f-9b37-4819-9d6d-e9022a2537fd-maas-api-service-ca\") pod \"cc0d564f-9b37-4819-9d6d-e9022a2537fd\" (UID: \"cc0d564f-9b37-4819-9d6d-e9022a2537fd\") "
Jul 17 14:43:48.561911 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:48.561890    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0d564f-9b37-4819-9d6d-e9022a2537fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc0d564f-9b37-4819-9d6d-e9022a2537fd" (UID: "cc0d564f-9b37-4819-9d6d-e9022a2537fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:43:48.662652 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:48.662577    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d564f-9b37-4819-9d6d-e9022a2537fd-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:43:49.335713 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:49.335682    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"
Jul 17 14:43:49.335903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:49.335725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl" event={"ID":"cc0d564f-9b37-4819-9d6d-e9022a2537fd","Type":"ContainerDied","Data":"76efcd12139b36fd7e4b7c2a68c6fffd3d5dfba3cb21547ddecb2d587932456b"}
Jul 17 14:43:49.335903 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:49.335783    2575 scope.go:117] "RemoveContainer" containerID="3c33af957f52b7e34362101d87e1a304c281583863e8381bad92c4f37c8463e7"
Jul 17 14:43:49.374842 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:49.374812    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"]
Jul 17 14:43:49.378913 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:49.378888    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-wrwtl"]
Jul 17 14:43:50.074839 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:50.074807    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" path="/var/lib/kubelet/pods/cc0d564f-9b37-4819-9d6d-e9022a2537fd/volumes"
Jul 17 14:43:57.069394 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:57.069363    2575 scope.go:117] "RemoveContainer" containerID="1de49b64cd97f2d0cdaa3c04db83bd3d3802125b4447a65f6ec48b2fe0e1573b"
Jul 17 14:43:57.378810 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:57.378782    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" event={"ID":"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0","Type":"ContainerStarted","Data":"e7eff55d14d0bcc23a3b7351d9c39d76e2df5aeb62f3808e11fb0bfdb9b9f5de"}
Jul 17 14:43:57.414172 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:57.414125    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" podStartSLOduration=15.414107345 podStartE2EDuration="15.414107345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:43:57.412573608 +0000 UTC m=+3515.799921047" watchObservedRunningTime="2026-07-17 14:43:57.414107345 +0000 UTC m=+3515.801454785"
Jul 17 14:43:58.384821 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:58.384788    2575 generic.go:356] "Generic (PLEG): container finished" podID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerID="e7eff55d14d0bcc23a3b7351d9c39d76e2df5aeb62f3808e11fb0bfdb9b9f5de" exitCode=6
Jul 17 14:43:58.385220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:58.384861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" event={"ID":"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0","Type":"ContainerDied","Data":"e7eff55d14d0bcc23a3b7351d9c39d76e2df5aeb62f3808e11fb0bfdb9b9f5de"}
Jul 17 14:43:58.385220 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:58.384921    2575 scope.go:117] "RemoveContainer" containerID="1de49b64cd97f2d0cdaa3c04db83bd3d3802125b4447a65f6ec48b2fe0e1573b"
Jul 17 14:43:58.385339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:58.385295    2575 scope.go:117] "RemoveContainer" containerID="e7eff55d14d0bcc23a3b7351d9c39d76e2df5aeb62f3808e11fb0bfdb9b9f5de"
Jul 17 14:43:58.385508 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:43:58.385485    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-cdrll_odh-ai-gateway-infra(4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0"
Jul 17 14:43:58.426761 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:58.426732    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"]
Jul 17 14:43:59.525272 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:59.525246    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"
Jul 17 14:43:59.555662 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:59.555635    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0-maas-api-service-ca\") pod \"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0\" (UID: \"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0\") "
Jul 17 14:43:59.555956 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:59.555936    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" (UID: "4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:43:59.657160 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:43:59.657082    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:00.397415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:00.397325    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll" event={"ID":"4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0","Type":"ContainerDied","Data":"9ff03fb7607dedfd035b47d82b0b8ce3086f22c92e1ae8ed51407f7d24703566"}
Jul 17 14:44:00.397415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:00.397390    2575 scope.go:117] "RemoveContainer" containerID="e7eff55d14d0bcc23a3b7351d9c39d76e2df5aeb62f3808e11fb0bfdb9b9f5de"
Jul 17 14:44:00.397664 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:00.397335    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"
Jul 17 14:44:00.415064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:00.415042    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"]
Jul 17 14:44:00.422581 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:00.422558    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-cdrll"]
Jul 17 14:44:02.075833 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:02.075799    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" path="/var/lib/kubelet/pods/4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0/volumes"
Jul 17 14:44:05.768237 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768194    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"]
Jul 17 14:44:05.768805 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768782    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:05.768894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768807    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:05.768894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768826    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:05.768894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768834    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:05.768894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768864    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:44:05.768894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768872    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768898    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768907    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768921    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.768929    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.769022    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.769036    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8cf2d6a-9a68-4615-b75d-66554bb3a86d" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.769049    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.769060    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.769069    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.769080    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:05.769140 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.769088    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ca73a7-925e-4751-82ca-6fb3c2ad27a2" containerName="revoke-keys"
Jul 17 14:44:05.773479 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.773458    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"
Jul 17 14:44:05.775768 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.775745    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:44:05.782914 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.782888    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"]
Jul 17 14:44:05.813648 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.813616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ae172b-635a-424c-87c4-fddb78994038-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m\" (UID: \"e6ae172b-635a-424c-87c4-fddb78994038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"
Jul 17 14:44:05.914783 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.914750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ae172b-635a-424c-87c4-fddb78994038-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m\" (UID: \"e6ae172b-635a-424c-87c4-fddb78994038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"
Jul 17 14:44:05.915342 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:05.915323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ae172b-635a-424c-87c4-fddb78994038-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m\" (UID: \"e6ae172b-635a-424c-87c4-fddb78994038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"
Jul 17 14:44:06.086481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:06.086403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"
Jul 17 14:44:06.222271 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:06.222242    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"]
Jul 17 14:44:06.224345 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:44:06.224319    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ae172b_635a_424c_87c4_fddb78994038.slice/crio-b72c42b71a069680d9c5efa4c7a94fd32470d3df55a74ef48c30011043ba8bb7 WatchSource:0}: Error finding container b72c42b71a069680d9c5efa4c7a94fd32470d3df55a74ef48c30011043ba8bb7: Status 404 returned error can't find the container with id b72c42b71a069680d9c5efa4c7a94fd32470d3df55a74ef48c30011043ba8bb7
Jul 17 14:44:06.430722 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:06.430641    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m" event={"ID":"e6ae172b-635a-424c-87c4-fddb78994038","Type":"ContainerStarted","Data":"b72c42b71a069680d9c5efa4c7a94fd32470d3df55a74ef48c30011043ba8bb7"}
Jul 17 14:44:07.436780 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:07.436699    2575 generic.go:356] "Generic (PLEG): container finished" podID="e6ae172b-635a-424c-87c4-fddb78994038" containerID="f7e53ed0507c775c2d39e3f2a8ffb71ac5e185a27c8ff482ef70e66c204ecdfd" exitCode=0
Jul 17 14:44:07.436780 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:07.436741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m" event={"ID":"e6ae172b-635a-424c-87c4-fddb78994038","Type":"ContainerDied","Data":"f7e53ed0507c775c2d39e3f2a8ffb71ac5e185a27c8ff482ef70e66c204ecdfd"}
Jul 17 14:44:08.577440 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:08.577416    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"
Jul 17 14:44:08.639151 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:08.639117    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ae172b-635a-424c-87c4-fddb78994038-maas-api-service-ca\") pod \"e6ae172b-635a-424c-87c4-fddb78994038\" (UID: \"e6ae172b-635a-424c-87c4-fddb78994038\") "
Jul 17 14:44:08.639573 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:08.639547    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ae172b-635a-424c-87c4-fddb78994038-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6ae172b-635a-424c-87c4-fddb78994038" (UID: "e6ae172b-635a-424c-87c4-fddb78994038"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:08.740715 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:08.740684    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ae172b-635a-424c-87c4-fddb78994038-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:09.446866 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:09.446839    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"
Jul 17 14:44:09.446866 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:09.446847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m" event={"ID":"e6ae172b-635a-424c-87c4-fddb78994038","Type":"ContainerDied","Data":"b72c42b71a069680d9c5efa4c7a94fd32470d3df55a74ef48c30011043ba8bb7"}
Jul 17 14:44:09.447110 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:09.446899    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72c42b71a069680d9c5efa4c7a94fd32470d3df55a74ef48c30011043ba8bb7"
Jul 17 14:44:13.212873 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.212841    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"]
Jul 17 14:44:13.213263 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213249    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:13.213323 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213265    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:13.213323 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213290    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:13.213323 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213298    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f406aa8-208b-4d4d-8fd9-2d7ee0b06ef0" containerName="revoke-keys"
Jul 17 14:44:13.213431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213331    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6ae172b-635a-424c-87c4-fddb78994038" containerName="revoke-keys"
Jul 17 14:44:13.213431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213337    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6ae172b-635a-424c-87c4-fddb78994038" containerName="revoke-keys"
Jul 17 14:44:13.213431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213397    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0d564f-9b37-4819-9d6d-e9022a2537fd" containerName="revoke-keys"
Jul 17 14:44:13.213431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.213404    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6ae172b-635a-424c-87c4-fddb78994038" containerName="revoke-keys"
Jul 17 14:44:13.216267 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.216252    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"
Jul 17 14:44:13.218431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.218405    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:44:13.223962 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.223940    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"]
Jul 17 14:44:13.282237 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.282207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86aaaef2-578d-4844-bfd2-ecf3b1295856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k\" (UID: \"86aaaef2-578d-4844-bfd2-ecf3b1295856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"
Jul 17 14:44:13.383018 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.382993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86aaaef2-578d-4844-bfd2-ecf3b1295856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k\" (UID: \"86aaaef2-578d-4844-bfd2-ecf3b1295856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"
Jul 17 14:44:13.383556 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.383540    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86aaaef2-578d-4844-bfd2-ecf3b1295856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k\" (UID: \"86aaaef2-578d-4844-bfd2-ecf3b1295856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"
Jul 17 14:44:13.528088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.528062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"
Jul 17 14:44:13.653508 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:13.653474    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"]
Jul 17 14:44:13.654780 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:44:13.654752    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86aaaef2_578d_4844_bfd2_ecf3b1295856.slice/crio-6cf9a3ec13cecb8cd26479bec1d54138dbaa320e7ebf0f821f2170ca2b71a144 WatchSource:0}: Error finding container 6cf9a3ec13cecb8cd26479bec1d54138dbaa320e7ebf0f821f2170ca2b71a144: Status 404 returned error can't find the container with id 6cf9a3ec13cecb8cd26479bec1d54138dbaa320e7ebf0f821f2170ca2b71a144
Jul 17 14:44:14.074906 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.074844    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"]
Jul 17 14:44:14.078367 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.078352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"
Jul 17 14:44:14.083639 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.083618    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"]
Jul 17 14:44:14.190297 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.190255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60e69c7-8558-46db-8d07-594f27879db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f\" (UID: \"d60e69c7-8558-46db-8d07-594f27879db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"
Jul 17 14:44:14.291302 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.291264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60e69c7-8558-46db-8d07-594f27879db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f\" (UID: \"d60e69c7-8558-46db-8d07-594f27879db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"
Jul 17 14:44:14.291847 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.291829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60e69c7-8558-46db-8d07-594f27879db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f\" (UID: \"d60e69c7-8558-46db-8d07-594f27879db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"
Jul 17 14:44:14.390077 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.390019    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"
Jul 17 14:44:14.474742 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.474711    2575 generic.go:356] "Generic (PLEG): container finished" podID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerID="993b56b5642a71e5023af05860a7099e851a529f9fdc84cffcfd45b932f747d3" exitCode=6
Jul 17 14:44:14.474875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.474792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" event={"ID":"86aaaef2-578d-4844-bfd2-ecf3b1295856","Type":"ContainerDied","Data":"993b56b5642a71e5023af05860a7099e851a529f9fdc84cffcfd45b932f747d3"}
Jul 17 14:44:14.474875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.474852    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" event={"ID":"86aaaef2-578d-4844-bfd2-ecf3b1295856","Type":"ContainerStarted","Data":"6cf9a3ec13cecb8cd26479bec1d54138dbaa320e7ebf0f821f2170ca2b71a144"}
Jul 17 14:44:14.475055 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.475040    2575 scope.go:117] "RemoveContainer" containerID="993b56b5642a71e5023af05860a7099e851a529f9fdc84cffcfd45b932f747d3"
Jul 17 14:44:14.518247 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:14.518221    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"]
Jul 17 14:44:14.519644 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:44:14.519621    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60e69c7_8558_46db_8d07_594f27879db1.slice/crio-5572e6c4dc5636b36090ca574fa0eeeaf5c4090f445b9b5e668fea6f6153f0d8 WatchSource:0}: Error finding container 5572e6c4dc5636b36090ca574fa0eeeaf5c4090f445b9b5e668fea6f6153f0d8: Status 404 returned error can't find the container with id 5572e6c4dc5636b36090ca574fa0eeeaf5c4090f445b9b5e668fea6f6153f0d8
Jul 17 14:44:15.480988 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.480960    2575 generic.go:356] "Generic (PLEG): container finished" podID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerID="ddc666b427ab290358a97639ce4765d5a386affe2095d30393e327d9e5457b46" exitCode=6
Jul 17 14:44:15.481386 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.481041    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" event={"ID":"86aaaef2-578d-4844-bfd2-ecf3b1295856","Type":"ContainerDied","Data":"ddc666b427ab290358a97639ce4765d5a386affe2095d30393e327d9e5457b46"}
Jul 17 14:44:15.481386 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.481087    2575 scope.go:117] "RemoveContainer" containerID="993b56b5642a71e5023af05860a7099e851a529f9fdc84cffcfd45b932f747d3"
Jul 17 14:44:15.481386 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.481321    2575 scope.go:117] "RemoveContainer" containerID="ddc666b427ab290358a97639ce4765d5a386affe2095d30393e327d9e5457b46"
Jul 17 14:44:15.481570 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:15.481550    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k_odh-ai-gateway-infra(86aaaef2-578d-4844-bfd2-ecf3b1295856)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856"
Jul 17 14:44:15.482741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.482723    2575 generic.go:356] "Generic (PLEG): container finished" podID="d60e69c7-8558-46db-8d07-594f27879db1" containerID="2dc92520507d21e824d3fba35fcac25f87303cf47c754056998aaf98754ecac7" exitCode=6
Jul 17 14:44:15.482825 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.482801    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" event={"ID":"d60e69c7-8558-46db-8d07-594f27879db1","Type":"ContainerDied","Data":"2dc92520507d21e824d3fba35fcac25f87303cf47c754056998aaf98754ecac7"}
Jul 17 14:44:15.482872 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.482847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" event={"ID":"d60e69c7-8558-46db-8d07-594f27879db1","Type":"ContainerStarted","Data":"5572e6c4dc5636b36090ca574fa0eeeaf5c4090f445b9b5e668fea6f6153f0d8"}
Jul 17 14:44:15.482922 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:15.482909    2575 scope.go:117] "RemoveContainer" containerID="2dc92520507d21e824d3fba35fcac25f87303cf47c754056998aaf98754ecac7"
Jul 17 14:44:16.275911 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.275879    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"]
Jul 17 14:44:16.276092 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.276072    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7" podUID="5c3bde99-f970-40ea-96c9-1e72ca54b482" containerName="maas-api" containerID="cri-o://c64bfedaf611683c3c9490d9bf90d6aa66d79c1f39d8a0b51515ae6ea97849d1" gracePeriod=30
Jul 17 14:44:16.490930 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.490900    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c3bde99-f970-40ea-96c9-1e72ca54b482" containerID="c64bfedaf611683c3c9490d9bf90d6aa66d79c1f39d8a0b51515ae6ea97849d1" exitCode=0
Jul 17 14:44:16.491316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.490941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7" event={"ID":"5c3bde99-f970-40ea-96c9-1e72ca54b482","Type":"ContainerDied","Data":"c64bfedaf611683c3c9490d9bf90d6aa66d79c1f39d8a0b51515ae6ea97849d1"}
Jul 17 14:44:16.492827 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.492804    2575 scope.go:117] "RemoveContainer" containerID="ddc666b427ab290358a97639ce4765d5a386affe2095d30393e327d9e5457b46"
Jul 17 14:44:16.493090 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:16.493066    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k_odh-ai-gateway-infra(86aaaef2-578d-4844-bfd2-ecf3b1295856)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856"
Jul 17 14:44:16.494527 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.494503    2575 generic.go:356] "Generic (PLEG): container finished" podID="d60e69c7-8558-46db-8d07-594f27879db1" containerID="360d088dd4d4f022445a9c8650dde5a686a18aa5d6113e0be62f1837b924cc93" exitCode=6
Jul 17 14:44:16.494602 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.494581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" event={"ID":"d60e69c7-8558-46db-8d07-594f27879db1","Type":"ContainerDied","Data":"360d088dd4d4f022445a9c8650dde5a686a18aa5d6113e0be62f1837b924cc93"}
Jul 17 14:44:16.494659 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.494631    2575 scope.go:117] "RemoveContainer" containerID="2dc92520507d21e824d3fba35fcac25f87303cf47c754056998aaf98754ecac7"
Jul 17 14:44:16.494754 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.494735    2575 scope.go:117] "RemoveContainer" containerID="360d088dd4d4f022445a9c8650dde5a686a18aa5d6113e0be62f1837b924cc93"
Jul 17 14:44:16.494959 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:16.494938    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f_odh-ai-gateway-infra(d60e69c7-8558-46db-8d07-594f27879db1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" podUID="d60e69c7-8558-46db-8d07-594f27879db1"
Jul 17 14:44:16.517184 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.517165    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:44:16.613716 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.613691    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqzl4\" (UniqueName: \"kubernetes.io/projected/5c3bde99-f970-40ea-96c9-1e72ca54b482-kube-api-access-mqzl4\") pod \"5c3bde99-f970-40ea-96c9-1e72ca54b482\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") "
Jul 17 14:44:16.613859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.613731    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3bde99-f970-40ea-96c9-1e72ca54b482-openshift-service-ca\") pod \"5c3bde99-f970-40ea-96c9-1e72ca54b482\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") "
Jul 17 14:44:16.613859 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.613782    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls\") pod \"5c3bde99-f970-40ea-96c9-1e72ca54b482\" (UID: \"5c3bde99-f970-40ea-96c9-1e72ca54b482\") "
Jul 17 14:44:16.614110 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.614082    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3bde99-f970-40ea-96c9-1e72ca54b482-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c3bde99-f970-40ea-96c9-1e72ca54b482" (UID: "5c3bde99-f970-40ea-96c9-1e72ca54b482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:16.615762 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.615740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c3bde99-f970-40ea-96c9-1e72ca54b482" (UID: "5c3bde99-f970-40ea-96c9-1e72ca54b482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:44:16.615847 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.615738    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3bde99-f970-40ea-96c9-1e72ca54b482-kube-api-access-mqzl4" (OuterVolumeSpecName: "kube-api-access-mqzl4") pod "5c3bde99-f970-40ea-96c9-1e72ca54b482" (UID: "5c3bde99-f970-40ea-96c9-1e72ca54b482"). InnerVolumeSpecName "kube-api-access-mqzl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:44:16.714916 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.714895    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzl4\" (UniqueName: \"kubernetes.io/projected/5c3bde99-f970-40ea-96c9-1e72ca54b482-kube-api-access-mqzl4\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:16.714916 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.714915    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3bde99-f970-40ea-96c9-1e72ca54b482-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:16.715030 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:16.714924    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3bde99-f970-40ea-96c9-1e72ca54b482-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:17.502052 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:17.502024    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"
Jul 17 14:44:17.502540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:17.502017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7" event={"ID":"5c3bde99-f970-40ea-96c9-1e72ca54b482","Type":"ContainerDied","Data":"85adec5b0ba632483caf0565e39f8769538f0c5675cef3edfb46a8ae76310cbb"}
Jul 17 14:44:17.502540 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:17.502156    2575 scope.go:117] "RemoveContainer" containerID="c64bfedaf611683c3c9490d9bf90d6aa66d79c1f39d8a0b51515ae6ea97849d1"
Jul 17 14:44:17.503987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:17.503968    2575 scope.go:117] "RemoveContainer" containerID="360d088dd4d4f022445a9c8650dde5a686a18aa5d6113e0be62f1837b924cc93"
Jul 17 14:44:17.504158 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:17.504134    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f_odh-ai-gateway-infra(d60e69c7-8558-46db-8d07-594f27879db1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" podUID="d60e69c7-8558-46db-8d07-594f27879db1"
Jul 17 14:44:17.542377 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:17.542353    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"]
Jul 17 14:44:17.549603 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:17.549584    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-244a73-6675584657-vlft7"]
Jul 17 14:44:18.074691 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:18.074655    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3bde99-f970-40ea-96c9-1e72ca54b482" path="/var/lib/kubelet/pods/5c3bde99-f970-40ea-96c9-1e72ca54b482/volumes"
Jul 17 14:44:20.055027 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.054997    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"]
Jul 17 14:44:20.055431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.055417    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3bde99-f970-40ea-96c9-1e72ca54b482" containerName="maas-api"
Jul 17 14:44:20.055431 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.055433    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3bde99-f970-40ea-96c9-1e72ca54b482" containerName="maas-api"
Jul 17 14:44:20.055505 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.055495    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3bde99-f970-40ea-96c9-1e72ca54b482" containerName="maas-api"
Jul 17 14:44:20.060098 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.060080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"
Jul 17 14:44:20.066974 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.066946    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"]
Jul 17 14:44:20.141813 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.141782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04227f75-4e31-4e28-b9cb-f7f2111b48d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz\" (UID: \"04227f75-4e31-4e28-b9cb-f7f2111b48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"
Jul 17 14:44:20.243145 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.243114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04227f75-4e31-4e28-b9cb-f7f2111b48d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz\" (UID: \"04227f75-4e31-4e28-b9cb-f7f2111b48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"
Jul 17 14:44:20.243719 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.243695    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04227f75-4e31-4e28-b9cb-f7f2111b48d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz\" (UID: \"04227f75-4e31-4e28-b9cb-f7f2111b48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"
Jul 17 14:44:20.372579 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.372521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"
Jul 17 14:44:20.496765 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.496743    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"]
Jul 17 14:44:20.498803 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:44:20.498778    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04227f75_4e31_4e28_b9cb_f7f2111b48d3.slice/crio-df8cce2d78831dc5b44d6f28db820fa33225574083da142d58e1ec85535e308b WatchSource:0}: Error finding container df8cce2d78831dc5b44d6f28db820fa33225574083da142d58e1ec85535e308b: Status 404 returned error can't find the container with id df8cce2d78831dc5b44d6f28db820fa33225574083da142d58e1ec85535e308b
Jul 17 14:44:20.521454 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.521422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" event={"ID":"04227f75-4e31-4e28-b9cb-f7f2111b48d3","Type":"ContainerStarted","Data":"df8cce2d78831dc5b44d6f28db820fa33225574083da142d58e1ec85535e308b"}
Jul 17 14:44:20.872167 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.872133    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"]
Jul 17 14:44:20.876017 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:20.875992    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-244a73-l5b9m"]
Jul 17 14:44:21.527528 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:21.527502    2575 generic.go:356] "Generic (PLEG): container finished" podID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerID="521eed34b0fac9cd33a50806c60e65c3bc738f45b512901f6c9bc971964adaee" exitCode=6
Jul 17 14:44:21.527994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:21.527543    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" event={"ID":"04227f75-4e31-4e28-b9cb-f7f2111b48d3","Type":"ContainerDied","Data":"521eed34b0fac9cd33a50806c60e65c3bc738f45b512901f6c9bc971964adaee"}
Jul 17 14:44:21.527994 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:21.527797    2575 scope.go:117] "RemoveContainer" containerID="521eed34b0fac9cd33a50806c60e65c3bc738f45b512901f6c9bc971964adaee"
Jul 17 14:44:22.075713 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:22.075649    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6ae172b-635a-424c-87c4-fddb78994038" path="/var/lib/kubelet/pods/e6ae172b-635a-424c-87c4-fddb78994038/volumes"
Jul 17 14:44:22.533894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:22.533804    2575 generic.go:356] "Generic (PLEG): container finished" podID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerID="f731d8e68b682d24324e4059e68d1a3f8490a58cf20e6c15cc895454338165ea" exitCode=6
Jul 17 14:44:22.533894 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:22.533861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" event={"ID":"04227f75-4e31-4e28-b9cb-f7f2111b48d3","Type":"ContainerDied","Data":"f731d8e68b682d24324e4059e68d1a3f8490a58cf20e6c15cc895454338165ea"}
Jul 17 14:44:22.571676 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:22.533939    2575 scope.go:117] "RemoveContainer" containerID="521eed34b0fac9cd33a50806c60e65c3bc738f45b512901f6c9bc971964adaee"
Jul 17 14:44:22.571676 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:22.534074    2575 scope.go:117] "RemoveContainer" containerID="f731d8e68b682d24324e4059e68d1a3f8490a58cf20e6c15cc895454338165ea"
Jul 17 14:44:22.571676 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:22.534311    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz_odh-ai-gateway-infra(04227f75-4e31-4e28-b9cb-f7f2111b48d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3"
Jul 17 14:44:23.540607 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:23.540583    2575 scope.go:117] "RemoveContainer" containerID="f731d8e68b682d24324e4059e68d1a3f8490a58cf20e6c15cc895454338165ea"
Jul 17 14:44:23.540968 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:23.540748    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz_odh-ai-gateway-infra(04227f75-4e31-4e28-b9cb-f7f2111b48d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3"
Jul 17 14:44:27.069758 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:27.069732    2575 scope.go:117] "RemoveContainer" containerID="ddc666b427ab290358a97639ce4765d5a386affe2095d30393e327d9e5457b46"
Jul 17 14:44:27.562308 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:27.562265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" event={"ID":"86aaaef2-578d-4844-bfd2-ecf3b1295856","Type":"ContainerStarted","Data":"4b97df3a7056679c98aff1e888464817468e907ccffff029e0e792d13ba8a886"}
Jul 17 14:44:27.578535 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:27.578489    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" podStartSLOduration=14.578473982 podStartE2EDuration="14.578473982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:44:27.577680414 +0000 UTC m=+3545.965027855" watchObservedRunningTime="2026-07-17 14:44:27.578473982 +0000 UTC m=+3545.965821422"
Jul 17 14:44:28.567408 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.567371    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"]
Jul 17 14:44:28.569458 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.569431    2575 generic.go:356] "Generic (PLEG): container finished" podID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerID="4b97df3a7056679c98aff1e888464817468e907ccffff029e0e792d13ba8a886" exitCode=6
Jul 17 14:44:28.571213 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.571194    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" event={"ID":"86aaaef2-578d-4844-bfd2-ecf3b1295856","Type":"ContainerDied","Data":"4b97df3a7056679c98aff1e888464817468e907ccffff029e0e792d13ba8a886"}
Jul 17 14:44:28.571289 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.571233    2575 scope.go:117] "RemoveContainer" containerID="ddc666b427ab290358a97639ce4765d5a386affe2095d30393e327d9e5457b46"
Jul 17 14:44:28.571329 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.571313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"
Jul 17 14:44:28.571642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.571627    2575 scope.go:117] "RemoveContainer" containerID="4b97df3a7056679c98aff1e888464817468e907ccffff029e0e792d13ba8a886"
Jul 17 14:44:28.571844 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:28.571826    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k_odh-ai-gateway-infra(86aaaef2-578d-4844-bfd2-ecf3b1295856)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856"
Jul 17 14:44:28.589620 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.589598    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"]
Jul 17 14:44:28.592822 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.592801    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"]
Jul 17 14:44:28.717801 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.717765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5570c7d-fd8a-4c39-baa3-c198ed88ab76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5\" (UID: \"c5570c7d-fd8a-4c39-baa3-c198ed88ab76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"
Jul 17 14:44:28.819068 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.819008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5570c7d-fd8a-4c39-baa3-c198ed88ab76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5\" (UID: \"c5570c7d-fd8a-4c39-baa3-c198ed88ab76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"
Jul 17 14:44:28.819591 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.819576    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5570c7d-fd8a-4c39-baa3-c198ed88ab76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5\" (UID: \"c5570c7d-fd8a-4c39-baa3-c198ed88ab76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"
Jul 17 14:44:28.881421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:28.881398    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"
Jul 17 14:44:29.010022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:29.009994    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"]
Jul 17 14:44:29.012431 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:44:29.012405    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5570c7d_fd8a_4c39_baa3_c198ed88ab76.slice/crio-d901206dec8af409823a5b69eb067cabfac82b9fbd26f130719fcbd526bec3b3 WatchSource:0}: Error finding container d901206dec8af409823a5b69eb067cabfac82b9fbd26f130719fcbd526bec3b3: Status 404 returned error can't find the container with id d901206dec8af409823a5b69eb067cabfac82b9fbd26f130719fcbd526bec3b3
Jul 17 14:44:29.578339 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:29.578310    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5" event={"ID":"c5570c7d-fd8a-4c39-baa3-c198ed88ab76","Type":"ContainerStarted","Data":"b13014bb3988971cdfd83e0f796f4ba7371ba606918c23133ea7598485c4b76f"}
Jul 17 14:44:29.578698 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:29.578365    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5" event={"ID":"c5570c7d-fd8a-4c39-baa3-c198ed88ab76","Type":"ContainerStarted","Data":"d901206dec8af409823a5b69eb067cabfac82b9fbd26f130719fcbd526bec3b3"}
Jul 17 14:44:29.611626 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:29.611578    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5" podStartSLOduration=1.611562481 podStartE2EDuration="1.611562481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:44:29.609400486 +0000 UTC m=+3547.996747928" watchObservedRunningTime="2026-07-17 14:44:29.611562481 +0000 UTC m=+3547.998909920"
Jul 17 14:44:29.745976 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:29.745951    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"
Jul 17 14:44:29.928516 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:29.928486    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86aaaef2-578d-4844-bfd2-ecf3b1295856-maas-api-service-ca\") pod \"86aaaef2-578d-4844-bfd2-ecf3b1295856\" (UID: \"86aaaef2-578d-4844-bfd2-ecf3b1295856\") "
Jul 17 14:44:29.928803 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:29.928782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86aaaef2-578d-4844-bfd2-ecf3b1295856-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86aaaef2-578d-4844-bfd2-ecf3b1295856" (UID: "86aaaef2-578d-4844-bfd2-ecf3b1295856"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:30.029414 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.029385    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86aaaef2-578d-4844-bfd2-ecf3b1295856-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:30.586465 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.586436    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"
Jul 17 14:44:30.586917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.586479    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k" event={"ID":"86aaaef2-578d-4844-bfd2-ecf3b1295856","Type":"ContainerDied","Data":"6cf9a3ec13cecb8cd26479bec1d54138dbaa320e7ebf0f821f2170ca2b71a144"}
Jul 17 14:44:30.586917 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.586541    2575 scope.go:117] "RemoveContainer" containerID="4b97df3a7056679c98aff1e888464817468e907ccffff029e0e792d13ba8a886"
Jul 17 14:44:30.588062 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.588038    2575 generic.go:356] "Generic (PLEG): container finished" podID="c5570c7d-fd8a-4c39-baa3-c198ed88ab76" containerID="b13014bb3988971cdfd83e0f796f4ba7371ba606918c23133ea7598485c4b76f" exitCode=0
Jul 17 14:44:30.588177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.588111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5" event={"ID":"c5570c7d-fd8a-4c39-baa3-c198ed88ab76","Type":"ContainerDied","Data":"b13014bb3988971cdfd83e0f796f4ba7371ba606918c23133ea7598485c4b76f"}
Jul 17 14:44:30.603612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.603592    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"]
Jul 17 14:44:30.609936 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:30.609916    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-ws64k"]
Jul 17 14:44:31.069005 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:31.068978    2575 scope.go:117] "RemoveContainer" containerID="360d088dd4d4f022445a9c8650dde5a686a18aa5d6113e0be62f1837b924cc93"
Jul 17 14:44:31.595636 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:31.595607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" event={"ID":"d60e69c7-8558-46db-8d07-594f27879db1","Type":"ContainerStarted","Data":"c2799775cddf7b712c7ff142ef35cbfcac801ec68051b58552b22791f821a8a1"}
Jul 17 14:44:31.613470 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:31.613429    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" podStartSLOduration=17.613415175 podStartE2EDuration="17.613415175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:44:31.611557459 +0000 UTC m=+3549.998904909" watchObservedRunningTime="2026-07-17 14:44:31.613415175 +0000 UTC m=+3550.000762614"
Jul 17 14:44:31.745295 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:31.745255    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"
Jul 17 14:44:31.842976 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:31.842953    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5570c7d-fd8a-4c39-baa3-c198ed88ab76-maas-api-service-ca\") pod \"c5570c7d-fd8a-4c39-baa3-c198ed88ab76\" (UID: \"c5570c7d-fd8a-4c39-baa3-c198ed88ab76\") "
Jul 17 14:44:31.843319 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:31.843259    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5570c7d-fd8a-4c39-baa3-c198ed88ab76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5570c7d-fd8a-4c39-baa3-c198ed88ab76" (UID: "c5570c7d-fd8a-4c39-baa3-c198ed88ab76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:31.943627 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:31.943579    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5570c7d-fd8a-4c39-baa3-c198ed88ab76-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:32.080699 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.080661    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" path="/var/lib/kubelet/pods/86aaaef2-578d-4844-bfd2-ecf3b1295856/volumes"
Jul 17 14:44:32.473968 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.473938    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"]
Jul 17 14:44:32.474345 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474332    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:44:32.474396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474346    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:44:32.474396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474355    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5570c7d-fd8a-4c39-baa3-c198ed88ab76" containerName="revoke-keys"
Jul 17 14:44:32.474396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474360    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5570c7d-fd8a-4c39-baa3-c198ed88ab76" containerName="revoke-keys"
Jul 17 14:44:32.474396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474378    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:44:32.474396 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474385    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:44:32.474553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474446    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5570c7d-fd8a-4c39-baa3-c198ed88ab76" containerName="revoke-keys"
Jul 17 14:44:32.474553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474454    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:44:32.474553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474461    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:44:32.474553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.474466    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:44:32.477559 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.477542    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"
Jul 17 14:44:32.485690 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.485659    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"]
Jul 17 14:44:32.601998 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.601970    2575 generic.go:356] "Generic (PLEG): container finished" podID="d60e69c7-8558-46db-8d07-594f27879db1" containerID="c2799775cddf7b712c7ff142ef35cbfcac801ec68051b58552b22791f821a8a1" exitCode=6
Jul 17 14:44:32.602393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.602050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" event={"ID":"d60e69c7-8558-46db-8d07-594f27879db1","Type":"ContainerDied","Data":"c2799775cddf7b712c7ff142ef35cbfcac801ec68051b58552b22791f821a8a1"}
Jul 17 14:44:32.602393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.602114    2575 scope.go:117] "RemoveContainer" containerID="360d088dd4d4f022445a9c8650dde5a686a18aa5d6113e0be62f1837b924cc93"
Jul 17 14:44:32.602393 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.602362    2575 scope.go:117] "RemoveContainer" containerID="c2799775cddf7b712c7ff142ef35cbfcac801ec68051b58552b22791f821a8a1"
Jul 17 14:44:32.602600 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:32.602578    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f_odh-ai-gateway-infra(d60e69c7-8558-46db-8d07-594f27879db1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" podUID="d60e69c7-8558-46db-8d07-594f27879db1"
Jul 17 14:44:32.603477 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.603464    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"
Jul 17 14:44:32.603570 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.603465    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5" event={"ID":"c5570c7d-fd8a-4c39-baa3-c198ed88ab76","Type":"ContainerDied","Data":"d901206dec8af409823a5b69eb067cabfac82b9fbd26f130719fcbd526bec3b3"}
Jul 17 14:44:32.603612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.603592    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d901206dec8af409823a5b69eb067cabfac82b9fbd26f130719fcbd526bec3b3"
Jul 17 14:44:32.623062 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.623042    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"]
Jul 17 14:44:32.649553 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.649531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e9e6c-1539-401c-b1ef-1fcc149e0a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-d28lx\" (UID: \"172e9e6c-1539-401c-b1ef-1fcc149e0a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"
Jul 17 14:44:32.750180 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.750122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e9e6c-1539-401c-b1ef-1fcc149e0a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-d28lx\" (UID: \"172e9e6c-1539-401c-b1ef-1fcc149e0a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"
Jul 17 14:44:32.750731 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.750711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e9e6c-1539-401c-b1ef-1fcc149e0a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-d28lx\" (UID: \"172e9e6c-1539-401c-b1ef-1fcc149e0a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"
Jul 17 14:44:32.789121 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.789102    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"
Jul 17 14:44:32.915064 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:32.915035    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"]
Jul 17 14:44:32.917870 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:44:32.917844    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172e9e6c_1539_401c_b1ef_1fcc149e0a3e.slice/crio-d82bd97669c287626b449e7c77d25dbd5cb17b78571424977f9d0646112afdc1 WatchSource:0}: Error finding container d82bd97669c287626b449e7c77d25dbd5cb17b78571424977f9d0646112afdc1: Status 404 returned error can't find the container with id d82bd97669c287626b449e7c77d25dbd5cb17b78571424977f9d0646112afdc1
Jul 17 14:44:33.609589 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.609564    2575 generic.go:356] "Generic (PLEG): container finished" podID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerID="c8df38ff1f1476d38b303b9c6be8c38b9b2e85aae692130f3f7ae1faa8f237c2" exitCode=6
Jul 17 14:44:33.609947 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.609646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" event={"ID":"172e9e6c-1539-401c-b1ef-1fcc149e0a3e","Type":"ContainerDied","Data":"c8df38ff1f1476d38b303b9c6be8c38b9b2e85aae692130f3f7ae1faa8f237c2"}
Jul 17 14:44:33.609947 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.609694    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" event={"ID":"172e9e6c-1539-401c-b1ef-1fcc149e0a3e","Type":"ContainerStarted","Data":"d82bd97669c287626b449e7c77d25dbd5cb17b78571424977f9d0646112afdc1"}
Jul 17 14:44:33.609947 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.609857    2575 scope.go:117] "RemoveContainer" containerID="c8df38ff1f1476d38b303b9c6be8c38b9b2e85aae692130f3f7ae1faa8f237c2"
Jul 17 14:44:33.761734 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.761713    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"
Jul 17 14:44:33.860399 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.860343    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60e69c7-8558-46db-8d07-594f27879db1-maas-api-service-ca\") pod \"d60e69c7-8558-46db-8d07-594f27879db1\" (UID: \"d60e69c7-8558-46db-8d07-594f27879db1\") "
Jul 17 14:44:33.860669 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.860650    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60e69c7-8558-46db-8d07-594f27879db1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d60e69c7-8558-46db-8d07-594f27879db1" (UID: "d60e69c7-8558-46db-8d07-594f27879db1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:33.961777 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:33.961754    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d60e69c7-8558-46db-8d07-594f27879db1-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:34.620100 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.620076    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"
Jul 17 14:44:34.620531 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.620120    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f" event={"ID":"d60e69c7-8558-46db-8d07-594f27879db1","Type":"ContainerDied","Data":"5572e6c4dc5636b36090ca574fa0eeeaf5c4090f445b9b5e668fea6f6153f0d8"}
Jul 17 14:44:34.620531 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.620176    2575 scope.go:117] "RemoveContainer" containerID="c2799775cddf7b712c7ff142ef35cbfcac801ec68051b58552b22791f821a8a1"
Jul 17 14:44:34.622484 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.622457    2575 generic.go:356] "Generic (PLEG): container finished" podID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerID="b2bbeec4f1a336a14c9c6e758bd7647cadd9c8ae9d47df04a373ab8b2a81998e" exitCode=6
Jul 17 14:44:34.622633 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.622557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" event={"ID":"172e9e6c-1539-401c-b1ef-1fcc149e0a3e","Type":"ContainerDied","Data":"b2bbeec4f1a336a14c9c6e758bd7647cadd9c8ae9d47df04a373ab8b2a81998e"}
Jul 17 14:44:34.622755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.622740    2575 scope.go:117] "RemoveContainer" containerID="b2bbeec4f1a336a14c9c6e758bd7647cadd9c8ae9d47df04a373ab8b2a81998e"
Jul 17 14:44:34.622925 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:34.622906    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-d28lx_odh-ai-gateway-infra(172e9e6c-1539-401c-b1ef-1fcc149e0a3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e"
Jul 17 14:44:34.628824 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.628806    2575 scope.go:117] "RemoveContainer" containerID="c8df38ff1f1476d38b303b9c6be8c38b9b2e85aae692130f3f7ae1faa8f237c2"
Jul 17 14:44:34.644911 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.644888    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"]
Jul 17 14:44:34.648946 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:34.648924    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-ltd2f"]
Jul 17 14:44:35.069260 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:35.069232    2575 scope.go:117] "RemoveContainer" containerID="f731d8e68b682d24324e4059e68d1a3f8490a58cf20e6c15cc895454338165ea"
Jul 17 14:44:35.628823 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:35.628797    2575 scope.go:117] "RemoveContainer" containerID="b2bbeec4f1a336a14c9c6e758bd7647cadd9c8ae9d47df04a373ab8b2a81998e"
Jul 17 14:44:35.629219 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:35.628988    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-d28lx_odh-ai-gateway-infra(172e9e6c-1539-401c-b1ef-1fcc149e0a3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e"
Jul 17 14:44:35.630371 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:35.630345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" event={"ID":"04227f75-4e31-4e28-b9cb-f7f2111b48d3","Type":"ContainerStarted","Data":"0cbd1d77d52740cdabd53038959abdd52731ab97e920b6a7c6b4875d55448c2a"}
Jul 17 14:44:35.662242 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:35.662195    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" podStartSLOduration=15.662182796 podStartE2EDuration="15.662182796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:44:35.660082583 +0000 UTC m=+3554.047430023" watchObservedRunningTime="2026-07-17 14:44:35.662182796 +0000 UTC m=+3554.049530263"
Jul 17 14:44:36.074517 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:36.074483    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d60e69c7-8558-46db-8d07-594f27879db1" path="/var/lib/kubelet/pods/d60e69c7-8558-46db-8d07-594f27879db1/volumes"
Jul 17 14:44:36.643508 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:36.643479    2575 generic.go:356] "Generic (PLEG): container finished" podID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerID="0cbd1d77d52740cdabd53038959abdd52731ab97e920b6a7c6b4875d55448c2a" exitCode=6
Jul 17 14:44:36.643927 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:36.643547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" event={"ID":"04227f75-4e31-4e28-b9cb-f7f2111b48d3","Type":"ContainerDied","Data":"0cbd1d77d52740cdabd53038959abdd52731ab97e920b6a7c6b4875d55448c2a"}
Jul 17 14:44:36.643927 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:36.643621    2575 scope.go:117] "RemoveContainer" containerID="f731d8e68b682d24324e4059e68d1a3f8490a58cf20e6c15cc895454338165ea"
Jul 17 14:44:36.643927 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:36.643847    2575 scope.go:117] "RemoveContainer" containerID="0cbd1d77d52740cdabd53038959abdd52731ab97e920b6a7c6b4875d55448c2a"
Jul 17 14:44:36.644084 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:36.644031    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz_odh-ai-gateway-infra(04227f75-4e31-4e28-b9cb-f7f2111b48d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3"
Jul 17 14:44:36.669811 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:36.669790    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"]
Jul 17 14:44:37.783379 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:37.783354    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"
Jul 17 14:44:37.891359 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:37.891269    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04227f75-4e31-4e28-b9cb-f7f2111b48d3-maas-api-service-ca\") pod \"04227f75-4e31-4e28-b9cb-f7f2111b48d3\" (UID: \"04227f75-4e31-4e28-b9cb-f7f2111b48d3\") "
Jul 17 14:44:37.891546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:37.891526    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04227f75-4e31-4e28-b9cb-f7f2111b48d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04227f75-4e31-4e28-b9cb-f7f2111b48d3" (UID: "04227f75-4e31-4e28-b9cb-f7f2111b48d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:37.891638 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:37.891627    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04227f75-4e31-4e28-b9cb-f7f2111b48d3-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:38.656417 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:38.656389    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"
Jul 17 14:44:38.656585 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:38.656434    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz" event={"ID":"04227f75-4e31-4e28-b9cb-f7f2111b48d3","Type":"ContainerDied","Data":"df8cce2d78831dc5b44d6f28db820fa33225574083da142d58e1ec85535e308b"}
Jul 17 14:44:38.656585 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:38.656496    2575 scope.go:117] "RemoveContainer" containerID="0cbd1d77d52740cdabd53038959abdd52731ab97e920b6a7c6b4875d55448c2a"
Jul 17 14:44:38.672457 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:38.672432    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"]
Jul 17 14:44:38.679373 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:38.679351    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-9jkvz"]
Jul 17 14:44:39.076499 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.076473    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"]
Jul 17 14:44:39.076874 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.076621    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646" podUID="a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" containerName="maas-api" containerID="cri-o://cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572" gracePeriod=30
Jul 17 14:44:39.325053 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.325030    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:44:39.504577 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.504552    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-openshift-service-ca\") pod \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") "
Jul 17 14:44:39.504755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.504613    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbrhc\" (UniqueName: \"kubernetes.io/projected/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-kube-api-access-dbrhc\") pod \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") "
Jul 17 14:44:39.504755 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.504646    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls\") pod \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\" (UID: \"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1\") "
Jul 17 14:44:39.504945 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.504920    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" (UID: "a18c1d0e-e5c8-407e-9582-6e0bac26fbb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:39.506639 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.506611    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" (UID: "a18c1d0e-e5c8-407e-9582-6e0bac26fbb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 14:44:39.506741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.506665    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-kube-api-access-dbrhc" (OuterVolumeSpecName: "kube-api-access-dbrhc") pod "a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" (UID: "a18c1d0e-e5c8-407e-9582-6e0bac26fbb1"). InnerVolumeSpecName "kube-api-access-dbrhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 14:44:39.606128 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.606105    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrhc\" (UniqueName: \"kubernetes.io/projected/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-kube-api-access-dbrhc\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:39.606128 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.606127    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-maas-api-tls\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:39.606263 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.606136    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1-openshift-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:39.663320 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.663294    2575 generic.go:356] "Generic (PLEG): container finished" podID="a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" containerID="cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572" exitCode=0
Jul 17 14:44:39.663424 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.663337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646" event={"ID":"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1","Type":"ContainerDied","Data":"cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572"}
Jul 17 14:44:39.663424 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.663387    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646" event={"ID":"a18c1d0e-e5c8-407e-9582-6e0bac26fbb1","Type":"ContainerDied","Data":"5bdef7f707656b573cdc1e638a479c25321c8f6b753c2ac30a21d5ffd786eee8"}
Jul 17 14:44:39.663424 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.663410    2575 scope.go:117] "RemoveContainer" containerID="cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572"
Jul 17 14:44:39.663579 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.663344    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"
Jul 17 14:44:39.673120 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.673105    2575 scope.go:117] "RemoveContainer" containerID="cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572"
Jul 17 14:44:39.673393 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:39.673372    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572\": container with ID starting with cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572 not found: ID does not exist" containerID="cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572"
Jul 17 14:44:39.673482 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.673398    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572"} err="failed to get container status \"cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572\": rpc error: code = NotFound desc = could not find container \"cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572\": container with ID starting with cfcb36a02d6d6d2fc94360f122fd847d90976bc8d29264c3a300695e3fca0572 not found: ID does not exist"
Jul 17 14:44:39.685870 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.685849    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"]
Jul 17 14:44:39.689756 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:39.689732    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9eb655-5cb8b6bcc7-jm646"]
Jul 17 14:44:40.074650 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:40.074625    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" path="/var/lib/kubelet/pods/04227f75-4e31-4e28-b9cb-f7f2111b48d3/volumes"
Jul 17 14:44:40.075009 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:40.074995    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" path="/var/lib/kubelet/pods/a18c1d0e-e5c8-407e-9582-6e0bac26fbb1/volumes"
Jul 17 14:44:43.626666 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:43.626629    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"]
Jul 17 14:44:43.629314 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:43.629290    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9eb655-ss2r5"]
Jul 17 14:44:44.075528 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:44.075497    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5570c7d-fd8a-4c39-baa3-c198ed88ab76" path="/var/lib/kubelet/pods/c5570c7d-fd8a-4c39-baa3-c198ed88ab76/volumes"
Jul 17 14:44:47.069558 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:47.069528    2575 scope.go:117] "RemoveContainer" containerID="b2bbeec4f1a336a14c9c6e758bd7647cadd9c8ae9d47df04a373ab8b2a81998e"
Jul 17 14:44:47.703811 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:47.703785    2575 generic.go:356] "Generic (PLEG): container finished" podID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerID="78a5a0af414a5f9bda6661fc20d2b17f88ad689817eaa76bf8e674173274b9e7" exitCode=6
Jul 17 14:44:47.703934 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:47.703863    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" event={"ID":"172e9e6c-1539-401c-b1ef-1fcc149e0a3e","Type":"ContainerDied","Data":"78a5a0af414a5f9bda6661fc20d2b17f88ad689817eaa76bf8e674173274b9e7"}
Jul 17 14:44:47.704000 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:47.703936    2575 scope.go:117] "RemoveContainer" containerID="b2bbeec4f1a336a14c9c6e758bd7647cadd9c8ae9d47df04a373ab8b2a81998e"
Jul 17 14:44:47.704316 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:47.704299    2575 scope.go:117] "RemoveContainer" containerID="78a5a0af414a5f9bda6661fc20d2b17f88ad689817eaa76bf8e674173274b9e7"
Jul 17 14:44:47.704550 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:44:47.704529    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-d28lx_odh-ai-gateway-infra(172e9e6c-1539-401c-b1ef-1fcc149e0a3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e"
Jul 17 14:44:48.731972 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:48.731943    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"]
Jul 17 14:44:48.858023 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:48.857995    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"
Jul 17 14:44:48.874141 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:48.874096    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e9e6c-1539-401c-b1ef-1fcc149e0a3e-maas-api-service-ca\") pod \"172e9e6c-1539-401c-b1ef-1fcc149e0a3e\" (UID: \"172e9e6c-1539-401c-b1ef-1fcc149e0a3e\") "
Jul 17 14:44:48.874523 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:48.874499    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172e9e6c-1539-401c-b1ef-1fcc149e0a3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "172e9e6c-1539-401c-b1ef-1fcc149e0a3e" (UID: "172e9e6c-1539-401c-b1ef-1fcc149e0a3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:44:48.975181 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:48.975149    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e9e6c-1539-401c-b1ef-1fcc149e0a3e-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:44:49.716800 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:49.716770    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"
Jul 17 14:44:49.716992 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:49.716773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx" event={"ID":"172e9e6c-1539-401c-b1ef-1fcc149e0a3e","Type":"ContainerDied","Data":"d82bd97669c287626b449e7c77d25dbd5cb17b78571424977f9d0646112afdc1"}
Jul 17 14:44:49.716992 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:49.716920    2575 scope.go:117] "RemoveContainer" containerID="78a5a0af414a5f9bda6661fc20d2b17f88ad689817eaa76bf8e674173274b9e7"
Jul 17 14:44:49.737249 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:49.737225    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"]
Jul 17 14:44:49.744459 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:49.744438    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-d28lx"]
Jul 17 14:44:50.074409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:50.074336    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" path="/var/lib/kubelet/pods/172e9e6c-1539-401c-b1ef-1fcc149e0a3e/volumes"
Jul 17 14:44:57.415420 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:57.415387    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f484f85fb-rj6cw_6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1/manager/0.log"
Jul 17 14:44:57.918195 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:57.918170    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57f79678b4-h4wpw_837e5de8-0d6c-47c9-a22d-a684cefaca39/copy-manifests/0.log"
Jul 17 14:44:57.925208 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:57.925185    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57f79678b4-h4wpw_837e5de8-0d6c-47c9-a22d-a684cefaca39/manager/0.log"
Jul 17 14:44:58.447421 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:58.447396    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75c9748d6-j459j_ac7c627d-7071-4c42-beb8-8ed0acae9279/maas-api/0.log"
Jul 17 14:44:58.823935 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:58.823910    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738310-7t5cd_0206d1e4-855a-4da5-85d3-979274f00c22/cleanup/0.log"
Jul 17 14:44:58.944488 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:44:58.944468    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-z2njc_d6050374-321f-48bb-aa8f-3d6040368eea/postgres/0.log"
Jul 17 14:45:00.146104 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146076    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz"]
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146453    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146464    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146473    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146478    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146485    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146490    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146508    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146516    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146532    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146537    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146543    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" containerName="maas-api"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146548    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" containerName="maas-api"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146554    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146560    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146566    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146571    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86aaaef2-578d-4844-bfd2-ecf3b1295856" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146576    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:00.146592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146581    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:00.147190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146651    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:00.147190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146658    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.147190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146663    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:00.147190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146670    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:00.147190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146676    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:00.147190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146681    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:00.147190 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.146687    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18c1d0e-e5c8-407e-9582-6e0bac26fbb1" containerName="maas-api"
Jul 17 14:45:00.150665 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.150650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz"
Jul 17 14:45:00.153090 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.153069    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfbc4\""
Jul 17 14:45:00.158438 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.158412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz"]
Jul 17 14:45:00.161179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.161164    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz"
Jul 17 14:45:00.288987 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.288965    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz"]
Jul 17 14:45:00.291590 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:45:00.291563    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c6aa29_eb45_4676_afd2_2e65c74cd3c7.slice/crio-8e7b2aec8f2fa9dda9dc7e994f48d851bc62ed3f4254800f9a078925a1173dbc WatchSource:0}: Error finding container 8e7b2aec8f2fa9dda9dc7e994f48d851bc62ed3f4254800f9a078925a1173dbc: Status 404 returned error can't find the container with id 8e7b2aec8f2fa9dda9dc7e994f48d851bc62ed3f4254800f9a078925a1173dbc
Jul 17 14:45:00.773521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.773445    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz" event={"ID":"15c6aa29-eb45-4676-afd2-2e65c74cd3c7","Type":"ContainerStarted","Data":"97fa41971339a33297f7c3b293608130163e5e5674fe3cb300569f4ebdc998a6"}
Jul 17 14:45:00.773521 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.773496    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz" event={"ID":"15c6aa29-eb45-4676-afd2-2e65c74cd3c7","Type":"ContainerStarted","Data":"8e7b2aec8f2fa9dda9dc7e994f48d851bc62ed3f4254800f9a078925a1173dbc"}
Jul 17 14:45:00.789883 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:00.789837    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz" podStartSLOduration=0.789823056 podStartE2EDuration="789.823056ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:45:00.788527413 +0000 UTC m=+3579.175874853" watchObservedRunningTime="2026-07-17 14:45:00.789823056 +0000 UTC m=+3579.177170552"
Jul 17 14:45:01.648784 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:01.648758    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-x4zlz_525694bd-6c3e-4648-81db-40546200adc4/discovery/0.log"
Jul 17 14:45:01.785333 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:01.785299    2575 generic.go:356] "Generic (PLEG): container finished" podID="15c6aa29-eb45-4676-afd2-2e65c74cd3c7" containerID="97fa41971339a33297f7c3b293608130163e5e5674fe3cb300569f4ebdc998a6" exitCode=0
Jul 17 14:45:01.785476 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:01.785363    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz" event={"ID":"15c6aa29-eb45-4676-afd2-2e65c74cd3c7","Type":"ContainerDied","Data":"97fa41971339a33297f7c3b293608130163e5e5674fe3cb300569f4ebdc998a6"}
Jul 17 14:45:02.142662 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:02.142633    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-lm4zz_0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da/payload-pre-processing/0.log"
Jul 17 14:45:02.263026 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:02.263001    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-j8rdn_55924f23-340f-407d-824d-ddff617a1f7a/payload-processing/0.log"
Jul 17 14:45:02.744720 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:02.744697    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-m6n2q_ec71b2e3-2930-4625-b4f3-9572a0f9ccbf/main/0.log"
Jul 17 14:45:02.915256 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:02.915235    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz"
Jul 17 14:45:02.983088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:02.983057    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sdrb8_bc89dbee-026b-4a0f-a3d8-3eb6a652f32d/main/0.log"
Jul 17 14:45:03.257660 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.257634    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"]
Jul 17 14:45:03.258040 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258027    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c6aa29-eb45-4676-afd2-2e65c74cd3c7" containerName="cleanup"
Jul 17 14:45:03.258088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258041    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c6aa29-eb45-4676-afd2-2e65c74cd3c7" containerName="cleanup"
Jul 17 14:45:03.258088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258065    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:03.258088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258073    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:03.258088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258081    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:03.258088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258087    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:03.258235 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258148    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="172e9e6c-1539-401c-b1ef-1fcc149e0a3e" containerName="revoke-keys"
Jul 17 14:45:03.258235 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258160    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c6aa29-eb45-4676-afd2-2e65c74cd3c7" containerName="cleanup"
Jul 17 14:45:03.258235 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258166    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="04227f75-4e31-4e28-b9cb-f7f2111b48d3" containerName="revoke-keys"
Jul 17 14:45:03.258235 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.258175    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d60e69c7-8558-46db-8d07-594f27879db1" containerName="revoke-keys"
Jul 17 14:45:03.261058 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.261043    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"
Jul 17 14:45:03.269134 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.269109    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"]
Jul 17 14:45:03.383539 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.383515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84755fac-af94-4c52-af5e-34f08c11fd55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-4448x\" (UID: \"84755fac-af94-4c52-af5e-34f08c11fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"
Jul 17 14:45:03.484126 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.484103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84755fac-af94-4c52-af5e-34f08c11fd55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-4448x\" (UID: \"84755fac-af94-4c52-af5e-34f08c11fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"
Jul 17 14:45:03.484677 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.484659    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84755fac-af94-4c52-af5e-34f08c11fd55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-398a9b19-4448x\" (UID: \"84755fac-af94-4c52-af5e-34f08c11fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"
Jul 17 14:45:03.572625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.572575    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"
Jul 17 14:45:03.796022 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.795998    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz"
Jul 17 14:45:03.796372 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.796036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738325-xwgzz" event={"ID":"15c6aa29-eb45-4676-afd2-2e65c74cd3c7","Type":"ContainerDied","Data":"8e7b2aec8f2fa9dda9dc7e994f48d851bc62ed3f4254800f9a078925a1173dbc"}
Jul 17 14:45:03.796372 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.796080    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7b2aec8f2fa9dda9dc7e994f48d851bc62ed3f4254800f9a078925a1173dbc"
Jul 17 14:45:03.905016 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:03.904960    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"]
Jul 17 14:45:03.908233 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:45:03.908207    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84755fac_af94_4c52_af5e_34f08c11fd55.slice/crio-195cf72f4364d1bea156263c20d74a39b557aa6bb5b4a33026b7b172c9d6900f WatchSource:0}: Error finding container 195cf72f4364d1bea156263c20d74a39b557aa6bb5b4a33026b7b172c9d6900f: Status 404 returned error can't find the container with id 195cf72f4364d1bea156263c20d74a39b557aa6bb5b4a33026b7b172c9d6900f
Jul 17 14:45:04.802296 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:04.802245    2575 generic.go:356] "Generic (PLEG): container finished" podID="84755fac-af94-4c52-af5e-34f08c11fd55" containerID="8feaf8f0de8c5203f51a2788432baf434b89a5614ddf1a7dce982f93453fe159" exitCode=6
Jul 17 14:45:04.802612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:04.802298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" event={"ID":"84755fac-af94-4c52-af5e-34f08c11fd55","Type":"ContainerDied","Data":"8feaf8f0de8c5203f51a2788432baf434b89a5614ddf1a7dce982f93453fe159"}
Jul 17 14:45:04.802612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:04.802357    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" event={"ID":"84755fac-af94-4c52-af5e-34f08c11fd55","Type":"ContainerStarted","Data":"195cf72f4364d1bea156263c20d74a39b557aa6bb5b4a33026b7b172c9d6900f"}
Jul 17 14:45:04.802612 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:04.802487    2575 scope.go:117] "RemoveContainer" containerID="8feaf8f0de8c5203f51a2788432baf434b89a5614ddf1a7dce982f93453fe159"
Jul 17 14:45:05.808621 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:05.808594    2575 generic.go:356] "Generic (PLEG): container finished" podID="84755fac-af94-4c52-af5e-34f08c11fd55" containerID="b7bab0a25ed6e946d701b22b09e9658fadaae8ed9e35767d970a3a7a93e7f79b" exitCode=6
Jul 17 14:45:05.809070 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:05.808630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" event={"ID":"84755fac-af94-4c52-af5e-34f08c11fd55","Type":"ContainerDied","Data":"b7bab0a25ed6e946d701b22b09e9658fadaae8ed9e35767d970a3a7a93e7f79b"}
Jul 17 14:45:05.809070 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:05.808684    2575 scope.go:117] "RemoveContainer" containerID="8feaf8f0de8c5203f51a2788432baf434b89a5614ddf1a7dce982f93453fe159"
Jul 17 14:45:05.809070 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:05.808917    2575 scope.go:117] "RemoveContainer" containerID="b7bab0a25ed6e946d701b22b09e9658fadaae8ed9e35767d970a3a7a93e7f79b"
Jul 17 14:45:05.809241 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:05.809144    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-4448x_odh-ai-gateway-infra(84755fac-af94-4c52-af5e-34f08c11fd55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" podUID="84755fac-af94-4c52-af5e-34f08c11fd55"
Jul 17 14:45:06.815576 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:06.815553    2575 scope.go:117] "RemoveContainer" containerID="b7bab0a25ed6e946d701b22b09e9658fadaae8ed9e35767d970a3a7a93e7f79b"
Jul 17 14:45:06.815942 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:06.815731    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-4448x_odh-ai-gateway-infra(84755fac-af94-4c52-af5e-34f08c11fd55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" podUID="84755fac-af94-4c52-af5e-34f08c11fd55"
Jul 17 14:45:10.118849 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.118816    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"]
Jul 17 14:45:10.122293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.122263    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"
Jul 17 14:45:10.132511 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.132488    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"]
Jul 17 14:45:10.236830 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.236804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df02448c-4c96-4ab3-92cc-f750f4b67618-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn\" (UID: \"df02448c-4c96-4ab3-92cc-f750f4b67618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"
Jul 17 14:45:10.337538 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.337514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df02448c-4c96-4ab3-92cc-f750f4b67618-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn\" (UID: \"df02448c-4c96-4ab3-92cc-f750f4b67618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"
Jul 17 14:45:10.338027 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.338013    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df02448c-4c96-4ab3-92cc-f750f4b67618-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn\" (UID: \"df02448c-4c96-4ab3-92cc-f750f4b67618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"
Jul 17 14:45:10.431863 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.431811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"
Jul 17 14:45:10.565311 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.565269    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"]
Jul 17 14:45:10.567484 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:45:10.567457    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf02448c_4c96_4ab3_92cc_f750f4b67618.slice/crio-f242a44f815959b8c1704e4d3f065653441b235dabf1746ddb6ec998762f2b72 WatchSource:0}: Error finding container f242a44f815959b8c1704e4d3f065653441b235dabf1746ddb6ec998762f2b72: Status 404 returned error can't find the container with id f242a44f815959b8c1704e4d3f065653441b235dabf1746ddb6ec998762f2b72
Jul 17 14:45:10.838338 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.838313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" event={"ID":"df02448c-4c96-4ab3-92cc-f750f4b67618","Type":"ContainerStarted","Data":"e87495fd818c375b66f8bc2cb86b0bf33bd6f2bdf160e9d0d461d7eb10c91010"}
Jul 17 14:45:10.838435 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.838358    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" event={"ID":"df02448c-4c96-4ab3-92cc-f750f4b67618","Type":"ContainerStarted","Data":"f242a44f815959b8c1704e4d3f065653441b235dabf1746ddb6ec998762f2b72"}
Jul 17 14:45:10.858289 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:10.858242    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" podStartSLOduration=0.858228572 podStartE2EDuration="858.228572ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:45:10.855766477 +0000 UTC m=+3589.243113928" watchObservedRunningTime="2026-07-17 14:45:10.858228572 +0000 UTC m=+3589.245576012"
Jul 17 14:45:11.843844 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:11.843815    2575 generic.go:356] "Generic (PLEG): container finished" podID="df02448c-4c96-4ab3-92cc-f750f4b67618" containerID="e87495fd818c375b66f8bc2cb86b0bf33bd6f2bdf160e9d0d461d7eb10c91010" exitCode=6
Jul 17 14:45:11.844259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:11.843878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" event={"ID":"df02448c-4c96-4ab3-92cc-f750f4b67618","Type":"ContainerDied","Data":"e87495fd818c375b66f8bc2cb86b0bf33bd6f2bdf160e9d0d461d7eb10c91010"}
Jul 17 14:45:11.844259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:11.844037    2575 scope.go:117] "RemoveContainer" containerID="e87495fd818c375b66f8bc2cb86b0bf33bd6f2bdf160e9d0d461d7eb10c91010"
Jul 17 14:45:12.760432 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:12.760404    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hq7zk_2d5310a1-8175-4478-b1c4-28ca2399c86b/global-pull-secret-syncer/0.log"
Jul 17 14:45:12.850618 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:12.850591    2575 generic.go:356] "Generic (PLEG): container finished" podID="df02448c-4c96-4ab3-92cc-f750f4b67618" containerID="52b5f38862f92a9a2d2bc3ece6ac182675fc6c8c8c9005b837720066638531c7" exitCode=6
Jul 17 14:45:12.850959 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:12.850628    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" event={"ID":"df02448c-4c96-4ab3-92cc-f750f4b67618","Type":"ContainerDied","Data":"52b5f38862f92a9a2d2bc3ece6ac182675fc6c8c8c9005b837720066638531c7"}
Jul 17 14:45:12.850959 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:12.850742    2575 scope.go:117] "RemoveContainer" containerID="e87495fd818c375b66f8bc2cb86b0bf33bd6f2bdf160e9d0d461d7eb10c91010"
Jul 17 14:45:12.851049 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:12.850985    2575 scope.go:117] "RemoveContainer" containerID="52b5f38862f92a9a2d2bc3ece6ac182675fc6c8c8c9005b837720066638531c7"
Jul 17 14:45:12.851181 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:12.851160    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn_odh-ai-gateway-infra(df02448c-4c96-4ab3-92cc-f750f4b67618)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" podUID="df02448c-4c96-4ab3-92cc-f750f4b67618"
Jul 17 14:45:12.928534 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:12.928511    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t47gn_b3405f65-f7c9-4913-8d29-611a74961d64/konnectivity-agent/0.log"
Jul 17 14:45:12.952551 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:12.952533    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-191.ec2.internal_40f4b672f95d6ba53b17d4bc79ab9528/haproxy/0.log"
Jul 17 14:45:13.857606 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:13.857576    2575 scope.go:117] "RemoveContainer" containerID="52b5f38862f92a9a2d2bc3ece6ac182675fc6c8c8c9005b837720066638531c7"
Jul 17 14:45:13.857959 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:13.857731    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn_odh-ai-gateway-infra(df02448c-4c96-4ab3-92cc-f750f4b67618)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" podUID="df02448c-4c96-4ab3-92cc-f750f4b67618"
Jul 17 14:45:14.127505 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.127439    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"]
Jul 17 14:45:14.131973 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.131958    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"
Jul 17 14:45:14.141237 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.141203    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"]
Jul 17 14:45:14.167592 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.167566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50776efe-9a9f-4217-976e-b300249cdf55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-24cjw\" (UID: \"50776efe-9a9f-4217-976e-b300249cdf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"
Jul 17 14:45:14.268219 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.268191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50776efe-9a9f-4217-976e-b300249cdf55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-24cjw\" (UID: \"50776efe-9a9f-4217-976e-b300249cdf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"
Jul 17 14:45:14.268748 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.268732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50776efe-9a9f-4217-976e-b300249cdf55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-988d1f69-24cjw\" (UID: \"50776efe-9a9f-4217-976e-b300249cdf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"
Jul 17 14:45:14.442674 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.442602    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"
Jul 17 14:45:14.575428 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.575405    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"]
Jul 17 14:45:14.577455 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:45:14.577409    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50776efe_9a9f_4217_976e_b300249cdf55.slice/crio-d675c1c518951b9ffc126672094823c845bae71d39bab08db2c7a13f00ff4702 WatchSource:0}: Error finding container d675c1c518951b9ffc126672094823c845bae71d39bab08db2c7a13f00ff4702: Status 404 returned error can't find the container with id d675c1c518951b9ffc126672094823c845bae71d39bab08db2c7a13f00ff4702
Jul 17 14:45:14.863980 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.863954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" event={"ID":"50776efe-9a9f-4217-976e-b300249cdf55","Type":"ContainerStarted","Data":"2f9e7168c5c9ee2bc186bbaa2c81d7708ef126978c60ccb2fcaa034a0b4d050e"}
Jul 17 14:45:14.864322 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.864013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" event={"ID":"50776efe-9a9f-4217-976e-b300249cdf55","Type":"ContainerStarted","Data":"d675c1c518951b9ffc126672094823c845bae71d39bab08db2c7a13f00ff4702"}
Jul 17 14:45:14.881123 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:14.881081    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" podStartSLOduration=0.881066197 podStartE2EDuration="881.066197ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 14:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:45:14.879801088 +0000 UTC m=+3593.267148528" watchObservedRunningTime="2026-07-17 14:45:14.881066197 +0000 UTC m=+3593.268413636"
Jul 17 14:45:15.869891 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:15.869858    2575 generic.go:356] "Generic (PLEG): container finished" podID="50776efe-9a9f-4217-976e-b300249cdf55" containerID="2f9e7168c5c9ee2bc186bbaa2c81d7708ef126978c60ccb2fcaa034a0b4d050e" exitCode=6
Jul 17 14:45:15.870293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:15.869918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" event={"ID":"50776efe-9a9f-4217-976e-b300249cdf55","Type":"ContainerDied","Data":"2f9e7168c5c9ee2bc186bbaa2c81d7708ef126978c60ccb2fcaa034a0b4d050e"}
Jul 17 14:45:15.870293 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:15.870103    2575 scope.go:117] "RemoveContainer" containerID="2f9e7168c5c9ee2bc186bbaa2c81d7708ef126978c60ccb2fcaa034a0b4d050e"
Jul 17 14:45:16.876145 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:16.876111    2575 generic.go:356] "Generic (PLEG): container finished" podID="50776efe-9a9f-4217-976e-b300249cdf55" containerID="74a4a50f01506a8772b0433ee9c5e8073cca1b6ece4bc80cf1b78e4791abeca8" exitCode=6
Jul 17 14:45:16.876546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:16.876198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" event={"ID":"50776efe-9a9f-4217-976e-b300249cdf55","Type":"ContainerDied","Data":"74a4a50f01506a8772b0433ee9c5e8073cca1b6ece4bc80cf1b78e4791abeca8"}
Jul 17 14:45:16.876546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:16.876252    2575 scope.go:117] "RemoveContainer" containerID="2f9e7168c5c9ee2bc186bbaa2c81d7708ef126978c60ccb2fcaa034a0b4d050e"
Jul 17 14:45:16.876546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:16.876427    2575 scope.go:117] "RemoveContainer" containerID="74a4a50f01506a8772b0433ee9c5e8073cca1b6ece4bc80cf1b78e4791abeca8"
Jul 17 14:45:16.876684 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:16.876615    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-24cjw_odh-ai-gateway-infra(50776efe-9a9f-4217-976e-b300249cdf55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" podUID="50776efe-9a9f-4217-976e-b300249cdf55"
Jul 17 14:45:17.882656 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:17.882622    2575 scope.go:117] "RemoveContainer" containerID="74a4a50f01506a8772b0433ee9c5e8073cca1b6ece4bc80cf1b78e4791abeca8"
Jul 17 14:45:17.883083 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:17.882858    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-24cjw_odh-ai-gateway-infra(50776efe-9a9f-4217-976e-b300249cdf55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" podUID="50776efe-9a9f-4217-976e-b300249cdf55"
Jul 17 14:45:20.069139 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:20.069115    2575 scope.go:117] "RemoveContainer" containerID="b7bab0a25ed6e946d701b22b09e9658fadaae8ed9e35767d970a3a7a93e7f79b"
Jul 17 14:45:20.627028 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:20.626981    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/1.log"
Jul 17 14:45:20.699676 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:20.699649    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:45:20.899709 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:20.899641    2575 generic.go:356] "Generic (PLEG): container finished" podID="84755fac-af94-4c52-af5e-34f08c11fd55" containerID="eb8c5b8fab49d2746e216cb5dcc46650b26a9649601fde450f45442855c7dcd9" exitCode=6
Jul 17 14:45:20.899835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:20.899715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" event={"ID":"84755fac-af94-4c52-af5e-34f08c11fd55","Type":"ContainerDied","Data":"eb8c5b8fab49d2746e216cb5dcc46650b26a9649601fde450f45442855c7dcd9"}
Jul 17 14:45:20.899835 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:20.899772    2575 scope.go:117] "RemoveContainer" containerID="b7bab0a25ed6e946d701b22b09e9658fadaae8ed9e35767d970a3a7a93e7f79b"
Jul 17 14:45:20.900059 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:20.900042    2575 scope.go:117] "RemoveContainer" containerID="eb8c5b8fab49d2746e216cb5dcc46650b26a9649601fde450f45442855c7dcd9"
Jul 17 14:45:20.900303 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:20.900220    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-398a9b19-4448x_odh-ai-gateway-infra(84755fac-af94-4c52-af5e-34f08c11fd55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" podUID="84755fac-af94-4c52-af5e-34f08c11fd55"
Jul 17 14:45:21.038639 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.038613    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8h5w_14444c67-24b3-4ae7-9e92-d20b7d012943/node-exporter/0.log"
Jul 17 14:45:21.075426 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.075401    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8h5w_14444c67-24b3-4ae7-9e92-d20b7d012943/kube-rbac-proxy/0.log"
Jul 17 14:45:21.100633 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.100613    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8h5w_14444c67-24b3-4ae7-9e92-d20b7d012943/init-textfile/0.log"
Jul 17 14:45:21.128642 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.128625    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-6tlwh_d6b3057c-b837-46ea-903f-f298702b6565/kube-rbac-proxy-main/0.log"
Jul 17 14:45:21.153990 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.153938    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-6tlwh_d6b3057c-b837-46ea-903f-f298702b6565/kube-rbac-proxy-self/0.log"
Jul 17 14:45:21.180590 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.180573    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-6tlwh_d6b3057c-b837-46ea-903f-f298702b6565/openshift-state-metrics/0.log"
Jul 17 14:45:21.219434 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.219416    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8/prometheus/0.log"
Jul 17 14:45:21.244091 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.244071    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8/config-reloader/0.log"
Jul 17 14:45:21.269672 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.269651    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8/thanos-sidecar/0.log"
Jul 17 14:45:21.294607 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.294593    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8/kube-rbac-proxy-web/0.log"
Jul 17 14:45:21.319310 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.319252    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8/kube-rbac-proxy/0.log"
Jul 17 14:45:21.346731 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.346715    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8/kube-rbac-proxy-thanos/0.log"
Jul 17 14:45:21.368972 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.368955    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b5dcc22-fbb2-4273-91c7-3f5e73bd03b8/init-config-reloader/0.log"
Jul 17 14:45:21.399681 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.399663    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-zdgts_0e300a68-c944-416c-9d7e-55ab8f3538c3/prometheus-operator/0.log"
Jul 17 14:45:21.423225 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.423157    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-zdgts_0e300a68-c944-416c-9d7e-55ab8f3538c3/kube-rbac-proxy/0.log"
Jul 17 14:45:21.558922 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.558897    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d688677-nvtnv_67811a6a-bfe3-41e5-a137-5bf62d77c126/thanos-query/0.log"
Jul 17 14:45:21.581203 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.581175    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d688677-nvtnv_67811a6a-bfe3-41e5-a137-5bf62d77c126/kube-rbac-proxy-web/0.log"
Jul 17 14:45:21.603415 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.603396    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d688677-nvtnv_67811a6a-bfe3-41e5-a137-5bf62d77c126/kube-rbac-proxy/0.log"
Jul 17 14:45:21.627238 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.627215    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d688677-nvtnv_67811a6a-bfe3-41e5-a137-5bf62d77c126/prom-label-proxy/0.log"
Jul 17 14:45:21.656179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.656153    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d688677-nvtnv_67811a6a-bfe3-41e5-a137-5bf62d77c126/kube-rbac-proxy-rules/0.log"
Jul 17 14:45:21.680314 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.680227    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d688677-nvtnv_67811a6a-bfe3-41e5-a137-5bf62d77c126/kube-rbac-proxy-metrics/0.log"
Jul 17 14:45:21.927856 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:21.927830    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"]
Jul 17 14:45:22.059613 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.059587    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"
Jul 17 14:45:22.125625 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.125599    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84755fac-af94-4c52-af5e-34f08c11fd55-maas-api-service-ca\") pod \"84755fac-af94-4c52-af5e-34f08c11fd55\" (UID: \"84755fac-af94-4c52-af5e-34f08c11fd55\") "
Jul 17 14:45:22.126040 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.125979    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84755fac-af94-4c52-af5e-34f08c11fd55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84755fac-af94-4c52-af5e-34f08c11fd55" (UID: "84755fac-af94-4c52-af5e-34f08c11fd55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:45:22.226498 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.226472    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84755fac-af94-4c52-af5e-34f08c11fd55-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:45:22.311739 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.311680    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:45:22.316786 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.316765    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:45:22.324594 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.324576    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:45:22.327290 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.327262    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7r6jb_ba0000c9-84a7-4f61-87e0-69d14de53c10/cluster-monitoring-operator/0.log"
Jul 17 14:45:22.331181 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.331161    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:45:22.338989 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.338960    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhqvc_13dc9407-48bb-4162-b663-f642fb0a6248/ovn-acl-logging/0.log"
Jul 17 14:45:22.527597 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.527573    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"]
Jul 17 14:45:22.527983 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.527968    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.528033 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.527985    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.528033 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.527999    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.528033 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.528004    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.528122 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.528078    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.528122 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.528088    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.528122 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.528095    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.531136 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.531122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"
Jul 17 14:45:22.537675 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.537650    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"]
Jul 17 14:45:22.628363 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.628306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81186e09-08c0-400b-9da2-44c41bac5a40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-sr4v8\" (UID: \"81186e09-08c0-400b-9da2-44c41bac5a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"
Jul 17 14:45:22.681454 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.681428    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"]
Jul 17 14:45:22.681804 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.681792    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.681853 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.681805    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" containerName="revoke-keys"
Jul 17 14:45:22.684806 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.684792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.687043 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.687021    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6g9mf\"/\"openshift-service-ca.crt\""
Jul 17 14:45:22.687043 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.687030    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6g9mf\"/\"kube-root-ca.crt\""
Jul 17 14:45:22.687184 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.687055    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6g9mf\"/\"default-dockercfg-fhjml\""
Jul 17 14:45:22.695404 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.695382    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"]
Jul 17 14:45:22.729708 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.729681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-lib-modules\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.729816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.729759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81186e09-08c0-400b-9da2-44c41bac5a40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-sr4v8\" (UID: \"81186e09-08c0-400b-9da2-44c41bac5a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"
Jul 17 14:45:22.729816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.729788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-proc\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.729816 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.729810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjl6\" (UniqueName: \"kubernetes.io/projected/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-kube-api-access-gpjl6\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.729961 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.729835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-sys\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.729961 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.729870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-podres\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.730333 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.730315    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81186e09-08c0-400b-9da2-44c41bac5a40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7874222-sr4v8\" (UID: \"81186e09-08c0-400b-9da2-44c41bac5a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"
Jul 17 14:45:22.830829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-lib-modules\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.830928 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-proc\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.830928 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjl6\" (UniqueName: \"kubernetes.io/projected/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-kube-api-access-gpjl6\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.830928 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-sys\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.831049 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-proc\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.831049 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-podres\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.831049 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830982    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-sys\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.831049 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.830997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-lib-modules\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.831169 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.831072    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-podres\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.840290 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.840259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjl6\" (UniqueName: \"kubernetes.io/projected/4c5dcb73-17e7-4f05-92b3-b9e0607d7add-kube-api-access-gpjl6\") pod \"perf-node-gather-daemonset-t54gx\" (UID: \"4c5dcb73-17e7-4f05-92b3-b9e0607d7add\") " pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:22.842088 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.842067    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"
Jul 17 14:45:22.913958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.913934    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"
Jul 17 14:45:22.914112 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.913966    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x" event={"ID":"84755fac-af94-4c52-af5e-34f08c11fd55","Type":"ContainerDied","Data":"195cf72f4364d1bea156263c20d74a39b557aa6bb5b4a33026b7b172c9d6900f"}
Jul 17 14:45:22.914112 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.914043    2575 scope.go:117] "RemoveContainer" containerID="eb8c5b8fab49d2746e216cb5dcc46650b26a9649601fde450f45442855c7dcd9"
Jul 17 14:45:22.945227 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.945201    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"]
Jul 17 14:45:22.952256 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.952229    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-398a9b19-4448x"]
Jul 17 14:45:22.983249 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.982929    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"]
Jul 17 14:45:22.986608 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:45:22.986580    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81186e09_08c0_400b_9da2_44c41bac5a40.slice/crio-0563e2802e6ff95d6c01975e9534c1e820495e4fcb39912bb766cce7ffa12d1f WatchSource:0}: Error finding container 0563e2802e6ff95d6c01975e9534c1e820495e4fcb39912bb766cce7ffa12d1f: Status 404 returned error can't find the container with id 0563e2802e6ff95d6c01975e9534c1e820495e4fcb39912bb766cce7ffa12d1f
Jul 17 14:45:22.994000 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:22.993982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:23.123509 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.123487    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"]
Jul 17 14:45:23.126089 ip-10-0-129-191 kubenswrapper[2575]: W0717 14:45:23.126063    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c5dcb73_17e7_4f05_92b3_b9e0607d7add.slice/crio-304dbe6a8b35e6b2185433ecaad98e3af5e22e079ee4e8b40f45fc83a015d1cc WatchSource:0}: Error finding container 304dbe6a8b35e6b2185433ecaad98e3af5e22e079ee4e8b40f45fc83a015d1cc: Status 404 returned error can't find the container with id 304dbe6a8b35e6b2185433ecaad98e3af5e22e079ee4e8b40f45fc83a015d1cc
Jul 17 14:45:23.514533 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.514504    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/2.log"
Jul 17 14:45:23.518872 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.518854    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmcnx_e8b7a27d-388f-4596-ad2b-bf20ad40b2a0/console-operator/3.log"
Jul 17 14:45:23.920489 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.920410    2575 generic.go:356] "Generic (PLEG): container finished" podID="81186e09-08c0-400b-9da2-44c41bac5a40" containerID="ca382fee6801b368a6c356081a75dca49d29afe813ae700966a06ad574169f8e" exitCode=6
Jul 17 14:45:23.920622 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.920500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" event={"ID":"81186e09-08c0-400b-9da2-44c41bac5a40","Type":"ContainerDied","Data":"ca382fee6801b368a6c356081a75dca49d29afe813ae700966a06ad574169f8e"}
Jul 17 14:45:23.920622 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.920598    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" event={"ID":"81186e09-08c0-400b-9da2-44c41bac5a40","Type":"ContainerStarted","Data":"0563e2802e6ff95d6c01975e9534c1e820495e4fcb39912bb766cce7ffa12d1f"}
Jul 17 14:45:23.920730 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.920642    2575 scope.go:117] "RemoveContainer" containerID="ca382fee6801b368a6c356081a75dca49d29afe813ae700966a06ad574169f8e"
Jul 17 14:45:23.923043 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.923013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx" event={"ID":"4c5dcb73-17e7-4f05-92b3-b9e0607d7add","Type":"ContainerStarted","Data":"cb716b4c90d4b726fac2bae320ac3f590d16c748769fe031881c8a66fa25aa57"}
Jul 17 14:45:23.923151 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.923058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx" event={"ID":"4c5dcb73-17e7-4f05-92b3-b9e0607d7add","Type":"ContainerStarted","Data":"304dbe6a8b35e6b2185433ecaad98e3af5e22e079ee4e8b40f45fc83a015d1cc"}
Jul 17 14:45:23.923151 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.923094    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:23.952179 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:23.952135    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx" podStartSLOduration=1.952123598 podStartE2EDuration="1.952123598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 14:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 14:45:23.950490404 +0000 UTC m=+3602.337837855" watchObservedRunningTime="2026-07-17 14:45:23.952123598 +0000 UTC m=+3602.339471036"
Jul 17 14:45:24.074641 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:24.074615    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84755fac-af94-4c52-af5e-34f08c11fd55" path="/var/lib/kubelet/pods/84755fac-af94-4c52-af5e-34f08c11fd55/volumes"
Jul 17 14:45:24.586958 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:24.586935    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-spqnn_9a868e7f-3f04-469f-8fe4-551bda323cf1/volume-data-source-validator/0.log"
Jul 17 14:45:24.929416 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:24.929383    2575 generic.go:356] "Generic (PLEG): container finished" podID="81186e09-08c0-400b-9da2-44c41bac5a40" containerID="411e8686bfcd746bb649157bd6cf04a5463423742366c0de5dd2231fddad1023" exitCode=6
Jul 17 14:45:24.929573 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:24.929464    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" event={"ID":"81186e09-08c0-400b-9da2-44c41bac5a40","Type":"ContainerDied","Data":"411e8686bfcd746bb649157bd6cf04a5463423742366c0de5dd2231fddad1023"}
Jul 17 14:45:24.929573 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:24.929525    2575 scope.go:117] "RemoveContainer" containerID="ca382fee6801b368a6c356081a75dca49d29afe813ae700966a06ad574169f8e"
Jul 17 14:45:24.929713 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:24.929697    2575 scope.go:117] "RemoveContainer" containerID="411e8686bfcd746bb649157bd6cf04a5463423742366c0de5dd2231fddad1023"
Jul 17 14:45:24.929964 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:24.929941    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-sr4v8_odh-ai-gateway-infra(81186e09-08c0-400b-9da2-44c41bac5a40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" podUID="81186e09-08c0-400b-9da2-44c41bac5a40"
Jul 17 14:45:25.635041 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:25.635015    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5x48l_2d686534-59bf-492c-89e5-98689ba395df/dns/0.log"
Jul 17 14:45:25.680759 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:25.680739    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5x48l_2d686534-59bf-492c-89e5-98689ba395df/kube-rbac-proxy/0.log"
Jul 17 14:45:25.827463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:25.827430    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58ggv_ef9b353f-aeb1-4d0b-adfb-01e472913c4b/dns-node-resolver/0.log"
Jul 17 14:45:25.935762 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:25.935697    2575 scope.go:117] "RemoveContainer" containerID="411e8686bfcd746bb649157bd6cf04a5463423742366c0de5dd2231fddad1023"
Jul 17 14:45:25.935876 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:25.935860    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-sr4v8_odh-ai-gateway-infra(81186e09-08c0-400b-9da2-44c41bac5a40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" podUID="81186e09-08c0-400b-9da2-44c41bac5a40"
Jul 17 14:45:26.069623 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:26.069600    2575 scope.go:117] "RemoveContainer" containerID="52b5f38862f92a9a2d2bc3ece6ac182675fc6c8c8c9005b837720066638531c7"
Jul 17 14:45:26.525614 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:26.525581    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gvl6k_c74d1625-9ecb-45cd-bc27-3f87ed8d40b3/node-ca/0.log"
Jul 17 14:45:26.942884 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:26.942859    2575 generic.go:356] "Generic (PLEG): container finished" podID="df02448c-4c96-4ab3-92cc-f750f4b67618" containerID="409e7689641f77d90530262231006b3c6123503a1ebb2c5b460f75c1c4e85dbd" exitCode=6
Jul 17 14:45:26.943197 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:26.942935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" event={"ID":"df02448c-4c96-4ab3-92cc-f750f4b67618","Type":"ContainerDied","Data":"409e7689641f77d90530262231006b3c6123503a1ebb2c5b460f75c1c4e85dbd"}
Jul 17 14:45:26.943197 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:26.942993    2575 scope.go:117] "RemoveContainer" containerID="52b5f38862f92a9a2d2bc3ece6ac182675fc6c8c8c9005b837720066638531c7"
Jul 17 14:45:26.943355 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:26.943220    2575 scope.go:117] "RemoveContainer" containerID="409e7689641f77d90530262231006b3c6123503a1ebb2c5b460f75c1c4e85dbd"
Jul 17 14:45:26.943489 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:26.943466    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn_odh-ai-gateway-infra(df02448c-4c96-4ab3-92cc-f750f4b67618)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" podUID="df02448c-4c96-4ab3-92cc-f750f4b67618"
Jul 17 14:45:27.703773 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:27.703744    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-x4zlz_525694bd-6c3e-4648-81db-40546200adc4/discovery/0.log"
Jul 17 14:45:27.861471 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:27.861441    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-lm4zz_0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da/payload-pre-processing/0.log"
Jul 17 14:45:27.893573 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:27.893551    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-j8rdn_55924f23-340f-407d-824d-ddff617a1f7a/payload-processing/0.log"
Jul 17 14:45:27.969563 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:27.969536    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"]
Jul 17 14:45:28.103453 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.103428    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"
Jul 17 14:45:28.173498 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.173471    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df02448c-4c96-4ab3-92cc-f750f4b67618-maas-api-service-ca\") pod \"df02448c-4c96-4ab3-92cc-f750f4b67618\" (UID: \"df02448c-4c96-4ab3-92cc-f750f4b67618\") "
Jul 17 14:45:28.173769 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.173748    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df02448c-4c96-4ab3-92cc-f750f4b67618-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df02448c-4c96-4ab3-92cc-f750f4b67618" (UID: "df02448c-4c96-4ab3-92cc-f750f4b67618"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:45:28.274777 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.274715    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df02448c-4c96-4ab3-92cc-f750f4b67618-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:45:28.664426 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.664362    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g7wdl_0e1a49fa-42d8-4f65-a8a2-7497d017e06d/serve-healthcheck-canary/0.log"
Jul 17 14:45:28.956481 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.956421    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"
Jul 17 14:45:28.956595 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.956413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn" event={"ID":"df02448c-4c96-4ab3-92cc-f750f4b67618","Type":"ContainerDied","Data":"f242a44f815959b8c1704e4d3f065653441b235dabf1746ddb6ec998762f2b72"}
Jul 17 14:45:28.956595 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.956561    2575 scope.go:117] "RemoveContainer" containerID="409e7689641f77d90530262231006b3c6123503a1ebb2c5b460f75c1c4e85dbd"
Jul 17 14:45:28.988298 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.985220    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"]
Jul 17 14:45:28.988298 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:28.988222    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fe1bea-4rsfn"]
Jul 17 14:45:29.068957 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.068939    2575 scope.go:117] "RemoveContainer" containerID="74a4a50f01506a8772b0433ee9c5e8073cca1b6ece4bc80cf1b78e4791abeca8"
Jul 17 14:45:29.244160 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.244140    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nrbn_e581b32b-7c8b-4268-a396-e910637ed696/kube-rbac-proxy/0.log"
Jul 17 14:45:29.267758 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.267740    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nrbn_e581b32b-7c8b-4268-a396-e910637ed696/exporter/0.log"
Jul 17 14:45:29.291706 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.291687    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nrbn_e581b32b-7c8b-4268-a396-e910637ed696/extractor/0.log"
Jul 17 14:45:29.938741 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.938686    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6g9mf/perf-node-gather-daemonset-t54gx"
Jul 17 14:45:29.965877 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.965846    2575 generic.go:356] "Generic (PLEG): container finished" podID="50776efe-9a9f-4217-976e-b300249cdf55" containerID="2431d2703dcd39870cddec391c7aac7247331a86e48cdbb1b86f175c89fe09f9" exitCode=6
Jul 17 14:45:29.966025 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.965925    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" event={"ID":"50776efe-9a9f-4217-976e-b300249cdf55","Type":"ContainerDied","Data":"2431d2703dcd39870cddec391c7aac7247331a86e48cdbb1b86f175c89fe09f9"}
Jul 17 14:45:29.966025 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.965990    2575 scope.go:117] "RemoveContainer" containerID="74a4a50f01506a8772b0433ee9c5e8073cca1b6ece4bc80cf1b78e4791abeca8"
Jul 17 14:45:29.966215 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:29.966201    2575 scope.go:117] "RemoveContainer" containerID="2431d2703dcd39870cddec391c7aac7247331a86e48cdbb1b86f175c89fe09f9"
Jul 17 14:45:29.966437 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:29.966416    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-988d1f69-24cjw_odh-ai-gateway-infra(50776efe-9a9f-4217-976e-b300249cdf55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" podUID="50776efe-9a9f-4217-976e-b300249cdf55"
Jul 17 14:45:30.075259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:30.075235    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df02448c-4c96-4ab3-92cc-f750f4b67618" path="/var/lib/kubelet/pods/df02448c-4c96-4ab3-92cc-f750f4b67618/volumes"
Jul 17 14:45:30.993186 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:30.993157    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"]
Jul 17 14:45:31.124076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:31.124051    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"
Jul 17 14:45:31.199513 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:31.199489    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50776efe-9a9f-4217-976e-b300249cdf55-maas-api-service-ca\") pod \"50776efe-9a9f-4217-976e-b300249cdf55\" (UID: \"50776efe-9a9f-4217-976e-b300249cdf55\") "
Jul 17 14:45:31.199829 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:31.199806    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50776efe-9a9f-4217-976e-b300249cdf55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50776efe-9a9f-4217-976e-b300249cdf55" (UID: "50776efe-9a9f-4217-976e-b300249cdf55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:45:31.300449 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:31.300395    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50776efe-9a9f-4217-976e-b300249cdf55-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:45:31.979709 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:31.979686    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"
Jul 17 14:45:31.979709 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:31.979691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw" event={"ID":"50776efe-9a9f-4217-976e-b300249cdf55","Type":"ContainerDied","Data":"d675c1c518951b9ffc126672094823c845bae71d39bab08db2c7a13f00ff4702"}
Jul 17 14:45:31.979875 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:31.979772    2575 scope.go:117] "RemoveContainer" containerID="2431d2703dcd39870cddec391c7aac7247331a86e48cdbb1b86f175c89fe09f9"
Jul 17 14:45:32.003096 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:32.003071    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"]
Jul 17 14:45:32.008259 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:32.008237    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-988d1f69-24cjw"]
Jul 17 14:45:32.075177 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:32.075158    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50776efe-9a9f-4217-976e-b300249cdf55" path="/var/lib/kubelet/pods/50776efe-9a9f-4217-976e-b300249cdf55/volumes"
Jul 17 14:45:32.245735 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:32.245706    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f484f85fb-rj6cw_6a91ab0e-eca8-4ca7-bb3c-a14cffaeccc1/manager/0.log"
Jul 17 14:45:32.479112 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:32.479090    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57f79678b4-h4wpw_837e5de8-0d6c-47c9-a22d-a684cefaca39/manager/0.log"
Jul 17 14:45:32.504945 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:32.504930    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57f79678b4-h4wpw_837e5de8-0d6c-47c9-a22d-a684cefaca39/copy-manifests/0.log"
Jul 17 14:45:33.362660 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:33.362631    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-x4zlz_525694bd-6c3e-4648-81db-40546200adc4/discovery/0.log"
Jul 17 14:45:33.502194 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:33.502169    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-lm4zz_0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da/payload-pre-processing/0.log"
Jul 17 14:45:33.521650 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:33.521628    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-j8rdn_55924f23-340f-407d-824d-ddff617a1f7a/payload-processing/0.log"
Jul 17 14:45:34.050170 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:34.050140    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77668bc899-lbr4h_cf0cedd5-727b-48a9-af08-e59b4b26f755/manager/0.log"
Jul 17 14:45:34.948397 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:34.948366    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-x4zlz_525694bd-6c3e-4648-81db-40546200adc4/discovery/0.log"
Jul 17 14:45:35.066613 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:35.066586    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-lm4zz_0c19cd24-ad9e-4e8c-9d86-c85a7d1a75da/payload-pre-processing/0.log"
Jul 17 14:45:35.088076 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:35.088051    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-j8rdn_55924f23-340f-407d-824d-ddff617a1f7a/payload-processing/0.log"
Jul 17 14:45:37.068720 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:37.068691    2575 scope.go:117] "RemoveContainer" containerID="411e8686bfcd746bb649157bd6cf04a5463423742366c0de5dd2231fddad1023"
Jul 17 14:45:38.010409 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:38.010383    2575 generic.go:356] "Generic (PLEG): container finished" podID="81186e09-08c0-400b-9da2-44c41bac5a40" containerID="73a9fe9cdc3a302b6cdec3ced0b9fe98459c5994dc51d22bbd65ca8d7e4f833a" exitCode=6
Jul 17 14:45:38.010546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:38.010423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" event={"ID":"81186e09-08c0-400b-9da2-44c41bac5a40","Type":"ContainerDied","Data":"73a9fe9cdc3a302b6cdec3ced0b9fe98459c5994dc51d22bbd65ca8d7e4f833a"}
Jul 17 14:45:38.010546 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:38.010477    2575 scope.go:117] "RemoveContainer" containerID="411e8686bfcd746bb649157bd6cf04a5463423742366c0de5dd2231fddad1023"
Jul 17 14:45:38.010774 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:38.010750    2575 scope.go:117] "RemoveContainer" containerID="73a9fe9cdc3a302b6cdec3ced0b9fe98459c5994dc51d22bbd65ca8d7e4f833a"
Jul 17 14:45:38.010986 ip-10-0-129-191 kubenswrapper[2575]: E0717 14:45:38.010967    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7874222-sr4v8_odh-ai-gateway-infra(81186e09-08c0-400b-9da2-44c41bac5a40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" podUID="81186e09-08c0-400b-9da2-44c41bac5a40"
Jul 17 14:45:39.042336 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.042304    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"]
Jul 17 14:45:39.179258 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.179228    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"
Jul 17 14:45:39.240463 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.240440    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-626wr_ef952a93-80dd-4ca5-9db3-61f802e93105/migrator/0.log"
Jul 17 14:45:39.259970 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.259942    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81186e09-08c0-400b-9da2-44c41bac5a40-maas-api-service-ca\") pod \"81186e09-08c0-400b-9da2-44c41bac5a40\" (UID: \"81186e09-08c0-400b-9da2-44c41bac5a40\") "
Jul 17 14:45:39.260322 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.260302    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81186e09-08c0-400b-9da2-44c41bac5a40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81186e09-08c0-400b-9da2-44c41bac5a40" (UID: "81186e09-08c0-400b-9da2-44c41bac5a40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 14:45:39.262005 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.261985    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-626wr_ef952a93-80dd-4ca5-9db3-61f802e93105/graceful-termination/0.log"
Jul 17 14:45:39.361529 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.361477    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81186e09-08c0-400b-9da2-44c41bac5a40-maas-api-service-ca\") on node \"ip-10-0-129-191.ec2.internal\" DevicePath \"\""
Jul 17 14:45:39.653869 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.653808    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-n5k7m_d6a6e949-2fd9-49fe-9d3d-7adc6e516721/kube-storage-version-migrator-operator/1.log"
Jul 17 14:45:39.654802 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:39.654784    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-n5k7m_d6a6e949-2fd9-49fe-9d3d-7adc6e516721/kube-storage-version-migrator-operator/0.log"
Jul 17 14:45:40.022582 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:40.022553    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8" event={"ID":"81186e09-08c0-400b-9da2-44c41bac5a40","Type":"ContainerDied","Data":"0563e2802e6ff95d6c01975e9534c1e820495e4fcb39912bb766cce7ffa12d1f"}
Jul 17 14:45:40.022765 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:40.022643    2575 scope.go:117] "RemoveContainer" containerID="73a9fe9cdc3a302b6cdec3ced0b9fe98459c5994dc51d22bbd65ca8d7e4f833a"
Jul 17 14:45:40.022765 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:40.022566    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"
Jul 17 14:45:40.044942 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:40.044916    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"]
Jul 17 14:45:40.047525 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:40.047494    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7874222-sr4v8"]
Jul 17 14:45:40.074562 ip-10-0-129-191 kubenswrapper[2575]: I0717 14:45:40.074541    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81186e09-08c0-400b-9da2-44c41bac5a40" path="/var/lib/kubelet/pods/81186e09-08c0-400b-9da2-44c41bac5a40/volumes"